diff --git a/docs/tickets/ticket-000023-5s-phase-1b-syllogism-synthesis-semiotics.md b/docs/tickets/ticket-000023-5s-phase-1b-syllogism-synthesis-semiotics.md index 7abf582..b48ac86 100644 --- a/docs/tickets/ticket-000023-5s-phase-1b-syllogism-synthesis-semiotics.md +++ b/docs/tickets/ticket-000023-5s-phase-1b-syllogism-synthesis-semiotics.md @@ -1,230 +1,392 @@ -# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics +# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics (carrier-aware) **Status:** open · awaiting go/no-go **Opened:** 2026-05-07 -**Scope:** Complete the 5S battery's three remaining sub-batteries — -Syllogism, Synthesis, Semiotics — that landed as zero-task stubs in -ticket #000021 Phase 1a. Each sub-battery gets a real fixture set -(seed → expand) and a runner that exercises arborist's actual -surface (claim-lattice parser, retrieval pipeline, label-swap -behavior). -**Audience:** fox + future bench authors + downstream v8 / v7-W work -that needs concrete fitness targets across all 5S dimensions. -**Hard constraint:** fixtures land deterministically (committed -JSONL with version pin). Bench-maxing discipline (5pp signal floor, -n≥3) applies. Each sub-battery is independently runnable via -`make bench-5s-`. Adding sub-batteries does NOT alter Phase 1a -behavior — Syntax + Semantics keep their existing fixture digests. +**Revised:** 2026-05-08 (cross-modality correction folded in per +review response) +**Scope:** Replace the three zero-task stubs in 5S +(Syllogism / Synthesis / Semiotics) with deterministic Phase-1b +runners + fixture sets. Implementation stays text / claim-lattice / +prose only; **the fixture schema is carrier-aware** so future +phases can attach visual, world-model, code, audio, sensor, and +hidden-channel-detection carriers without re-authoring the +fixture format. +**Audience:** fox + future bench authors + downstream v7-W / +multimodal work that needs 5S evaluation across non-text carriers. +**Hard constraint:** Phase 1a fixture digests stay pinned (Syntax ++ Semantics keep their `v1` digests). No schema bump, no +governance hash bump, no canonicalization-version bump. Unsupported +non-text carriers MUST fail or skip explicitly — never silently +accepted. No LLM-as-judge in any 5S runner. -**Source:** Dav1DPrometheus (BasementAGI host) named the 5Ss in his -authoritative framework; arborist honors the original wording. +**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ss: Syntax, +Semantics, Syllogism, Synthesis, Semiotics."* --- -## 1. Problem statement +## 1. Corrected purpose -Ticket #000021 Phase 1a shipped runners + seed fixtures for 5S -Syntax (`parse-pass`) and 5S Semantics (`equivalence`). The other -three sub-batteries — Syllogism, Synthesis, Semiotics — landed as -stubs that return zero-task `BatteryResult` objects so the full -suite at least runs. Stubs are visible in `bench/batteries/b_5s.py`. +Two coupled goals: -Without real implementations: +1. **Complete the 5S battery** — three stubs become real runners + with ≥30 deterministic fixtures each. +2. **Make 5S carrier-general** — Syntax, Semantics, Semiotics, + Syllogism, Synthesis are not prose-only. They are tests over + *sign-bearing representations*. Text is only the first carrier. -| Gap | Failure mode | +The architectural correction (folded in 2026-05-08): + +> **Syntax hidden in visual or alternate media is still syntax. +> Meaning hidden in non-text media is still semantics. A sign +> encoded in a non-text carrier is still semiotics.** + +#000023 lands text fixtures + text runners now. The fixture +**schema** must accommodate future cross-carrier work without +re-authoring. + +--- + +## 2. Carrier-aware 5S definitions + +| Sub-battery | Definition (carrier-general) | |---|---| -| **Syllogism** | No way to bench arborist's deductive-reasoning surface. v8 selection + SelfModel capability claims have no measurable target for "stepwise entailment validity." | -| **Synthesis** | No bench for "given a goal + a fact set, can the system produce a passing derivation?" — directly the claim-lattice + retrieval pipeline's output, which today only gets ad-hoc QA bench coverage. | -| **Semiotics** | Label-swap invariance under controlled symbol changes is unmeasured. Drift in proper-noun handling, code-name conventions, or alphabet shifts goes undetected. | +| **Syntax** | Formal arrangement rules of a carrier. Text: tokens, parse, markup. Code: AST, imports, indentation. Tables: rows/columns/headers. Images: layout, regions, visual grouping. World: object relations, spatial containment, event ordering. Audio: phoneme/order structure, timing. | +| **Semantics** | Invariant meaning preserved across representation changes. Text: synonym pairs, entity equivalence. Image: a stop sign and the text "STOP" carry equivalent traffic-control meaning. Chart: an upward trend and "X increases over time" carry equivalent data semantics. World: "A inside B" represented visually, symbolically, or as coordinates. | +| **Syllogism** | Stepwise valid deduction, given premises in any carrier. Text+table: "all rows with status=active have flag=true; row X has status=active; therefore X has flag=true." Image+rule: "A is inside B; rule says objects in B have property P; therefore A has P." World: "A left of B; B left of C; therefore A left of C." | +| **Synthesis** | Assembling multiple warranted elements into a coherent derivation. Text+chart: derive a claim only when textual and chart evidence agree. Image+OCR+metadata: synthesize while flagging carrier conflict. World-state: synthesize a temporal claim from multiple state observations. | +| **Semiotics** | Interpretation of signs, symbols, labels, references, frames, conventions, aliases, sign systems. Text: AMD↔"Advanced Micro Devices", Apple-fruit vs Apple-company, Orwell-frame vs literal-geography. Visual: red octagon = stop, skull icon = hazard, watermark = provenance, QR-pattern = encoded sign system, diagram-arrow = causality/flow/dependency. Future: visual steganographic carriers where surface differs from hidden encoding. | -### 1.1 Sub-battery semantics (per ticket #000021 §4.2) - -- **Syllogism:** each task is a proof chain + a step index + - expected step-valid pass/fail. Metric: step-validity-rate. -- **Synthesis:** each task is a goal description + a fact set + - expected derivation pass/fail. Metric: derivation-pass-rate. -- **Semiotics:** each task is a controlled label-swap + invariant - expected. Metric: invariance-under-swap. +Phase 1b implements only **text / claim-lattice / prose** carriers. +Future-carrier hooks land via fixture metadata (§3) that the v1 +runner ignores or rejects-cleanly. --- -## 2. Design choices +## 3. Fixture schema (carrier-aware) -### 2.1 Fixture sourcing for Syllogism +Common JSONL header per task: -**A. Hand-curated proof chains drawn from existing claim-lattice tests.** -arborist's `tests/test_claim_lattice.py` and `tests/test_directives.py` -already contain stepwise-claim examples. Promoting them to -fixtures gets us 30-50 entries cheaply. +```json +{ + "id": "5s--NNN", + "battery": "5s", + "sub_battery": "syllogism", + "version": "v1", -**B. Synthetic generators (template-based syllogisms).** Cheap; -templates need to be committed for reproducibility. + "carrier": "text", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "loss_report_refs": [], + "modality_notes": "Phase 1b: text-only; future carriers may include image/code/audio/world/hidden-channel.", -**Recommendation:** A first; add B in Phase 2 once the harness -stabilizes. - -### 2.2 Fixture sourcing for Synthesis - -**A. Ablate from existing QA fixtures.** Take questions from -`bench/qa_sweep.py` corpora; record which sources retrieve as the -fact set; mark expected derivation status. - -**B. Compose from claim-lattice docs.** Use docs already in shards; -build (goal, fact_set, derivation_expected) tuples by stripping -claims and asking whether they should re-derive. - -**Recommendation:** A — anchors against existing QA bench so -Synthesis-rate trends correlate with overall QA rate. - -### 2.3 Fixture sourcing for Semiotics - -**A. Hand-curated label-swap pairs.** Examples: `"AMD"↔"Advanced Micro -Devices"`, `"NYC"↔"New York City"`, code-name swaps inside -identical text. - -**B. Generated swaps via the concept-relations table.** arborist's -`concept_relations` already has synonym pairs from corpus -extraction; promote them as semiotic swap fixtures. - -**Recommendation:** B — leverages existing per-shard data, makes -fixtures regenerable from the corpus rather than hand-pinned. - -### 2.4 Runner integration - -Each sub-battery is a function in `bench/batteries/b_5s.py` -matching the existing `run_syntax`/`run_semantics` signature. -Replace the stub `_stub_result` returns with real implementations. -Existing test in `tests/test_bench_batteries.py -::test_stub_sub_batteries_return_zero` will need to flip to -non-zero assertions once these land. - ---- - -## 3. Recommendation - -Land in this order: - -1. **Syllogism** first — purely lexical (claim-lattice traversal), - no shard reads, fastest implementation. -2. **Semiotics** second — depends on `concept_relations` lookup - but no LLM call; deterministic. -3. **Synthesis** third — requires shard reads + retrieval pipeline; - slowest, biggest surface, highest variance. Defer until - Syllogism + Semiotics stabilize the harness. - -Phase 1b closure = all three sub-batteries shipping with at minimum -**30 deterministic fixtures each** and runners that integrate -cleanly into `make bench-5s`. - ---- - -## 4. Implementation sketch - -### 4.1 Fixture files - -``` -bench/fixtures/5s/syllogism-v1.jsonl # 30+ tasks -bench/fixtures/5s/synthesis-v1.jsonl # 30+ tasks -bench/fixtures/5s/semiotics-v1.jsonl # 30+ tasks + "task": { ... } +} ``` -Each fixture file carries the standard `_meta` head: +Field semantics: + +| Field | Phase 1b allowed values | Future values | +|---|---|---| +| `carrier` | `text`, `claim_lattice`, `prose` | `code`, `table`, `html`, `image`, `audio`, `layout`, `world`, `mixed`, `hidden_channel` | +| `domain` | `prose`, `wikitext`, `claim_lattice` | `python_ast`, `chart`, `spatial_state`, `propositional_logic`, etc. | +| `pi_star_ref` | `wikitext-base@v1`, `claim-lattice@v1` | future π* registry keys | +| `loss_report_refs` | `[]` | optional projection-loss record refs (#000022) | + +Unsupported `carrier` / `domain` values → runner fails task +explicitly with `reason="unsupported_carrier"` or skips with same +reason logged. Never silently accepted. + +--- + +## 4. Per-sub-battery implementation + +### 4.1 Syllogism + +**Metric:** `step_validity_rate = valid_steps / total_steps`. + +**Positive fixture:** + +```json +{ + "id": "5s-syllogism-001", + "carrier": "text", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "premises": [ + {"claim": "All mammals are animals.", "evidence_ids": ["E1"]}, + {"claim": "All dogs are mammals.", "evidence_ids": ["E2"]} + ], + "candidate_step": { + "claim": "All dogs are animals.", + "uses": ["E1", "E2"] + }, + "rule": "categorical_transitivity", + "expected": "pass" +} +``` + +**Negative fixture:** + +```json +{ + "id": "5s-syllogism-NEG-001", + "premises": [ + {"claim": "All dogs are animals.", "evidence_ids": ["E1"]}, + {"claim": "All cats are animals.", "evidence_ids": ["E2"]} + ], + "candidate_step": { + "claim": "All animals are dogs.", + "uses": ["E1", "E2"] + }, + "rule": "invalid_converse", + "expected": "fail" +} +``` + +**Runner behavior:** load fixture → parse premises into +claim-lattice → parse candidate step → check referenced prior +claims exist → check step is licensed by declared rule → return +pass/fail. Deterministic, no LLM judge. + +### 4.2 Synthesis + +**Metric:** `derivation_pass_rate = passed / total`. + +**Fixture shape:** + +```json +{ + "id": "5s-synthesis-001", + "carrier": "text", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "goal": "Derive whether the source supports the release-date claim.", + "fact_set": [ + {"id": "F1", "text": "Back to the Future was released on July 3, 1985."}, + {"id": "F2", "text": "The film was directed by Robert Zemeckis."} + ], + "expected_derivation": [ + {"claim": "Back to the Future was released on July 3, 1985.", "uses": ["F1"]} + ], + "expected": "pass" +} +``` + +**Future-carrier hook:** fact-set entries may later include +non-text carriers (`carrier: "chart"`, `carrier: "image"`). v1 +runner rejects them as unsupported. + +**Variance control:** if a Synthesis task uses arborist's retrieval +pipeline, the fixture meta MUST pin `shard_set`, `retrieval_mode`, +and `expected_fact_ids`. Bench-maxing 5pp signal floor handles +residual noise; pinned fact IDs eliminate the largest variance +source. + +### 4.3 Semiotics + +**Metric:** `invariance_under_swap = invariant_passes / total`. + +**Synonym-swap fixture (positive):** + +```json +{ + "id": "5s-semiotics-001", + "carrier": "text", + "domain": "prose", + "pi_star_ref": "wikitext-base@v1", + "source_text": "AMD released a new GPU.", + "swap": { + "from": "AMD", + "to": "Advanced Micro Devices", + "relation_kind": "synonym" + }, + "invariant_fields": ["entity", "predicate", "object"], + "expected_invariant": true +} +``` + +**Reference-frame fixture:** + +```json +{ + "id": "5s-semiotics-002", + "source_text": "Oceania has always been at war with Eastasia.", + "frame": "orwell_1984", + "literal_frame_expected": false, + "reference_frame_expected": true, + "expected_invariant": true +} +``` + +**Future-carrier hook (documented in fixture, not executed in v1):** + +```json +{ + "future_carrier_equivalent": { + "carrier": "image", + "description": "symbolic sign or visual glyph equivalent may later replace text label" + } +} +``` + +**Hidden-channel safety framing.** Phase 1b does NOT generate, +hide, or recover covert messages. Future hidden-channel fixtures +MUST be framed as **defensive detection / flagging** only: + +```json +{ + "carrier": "image", + "sub_battery": "semiotics", + "hidden_channel_expected": true, + "task_kind": "detect_unexpected_sign_channel", + "expected": "flag" +} +``` + +This keeps the benchmark defensive. The capability tested is +"can the system detect a suspicious carrier" — not "can the +system encode a covert message." + +**Source policy for v1 fixtures:** Semiotics fixtures use +`concept_relations.relation_kind = 'synonym'` only. Negative +fixtures pair with non-equivalent swaps explicitly tagged. + +--- + +## 5. Required file changes + +### 5.1 New fixture files + +``` +bench/fixtures/5s/syllogism-v1.jsonl # ≥30 deterministic tasks +bench/fixtures/5s/synthesis-v1.jsonl # ≥30 deterministic tasks +bench/fixtures/5s/semiotics-v1.jsonl # ≥30 deterministic tasks +``` + +Each carries the standard `_meta` head: ```json {"_meta":{"battery":"5s","sub_battery":"syllogism","version":"v1","task_count":30}} ``` -### 4.2 Runner additions +Phase 1a fixture digests (`syntax-v1.jsonl`, `semantics-v1.jsonl`) +stay pinned. -In `bench/batteries/b_5s.py`, replace stub bodies: +### 5.2 Runner functions -```python -def run_syllogism(fixtures_path: Path) -> BatteryResult: - # for each task: parse claim chain, walk step_index, assert - # the claim references valid evidence + the prior step entails - # this one under claim-lattice semantics. Metric: - # step_validity_rate = pass_count / (pass + fail). - ... -``` +In `bench/batteries/b_5s.py`, replace stub bodies of +`run_syllogism`, `run_synthesis`, `run_semiotics`. Existing +`run_syntax`, `run_semantics`, `_runtime_digest` stay untouched. -Same shape for `run_synthesis`, `run_semiotics`. Existing -`_runtime_digest` helper covers the runtime fingerprint. - -### 4.3 Makefile additions +### 5.3 Make targets ```makefile -bench-5s-syllogism: bootstrap +bench-5s-syllogism: ## 5S Syllogism (step-validity-rate) $(PY) -m bench.batteries.runner --battery 5s --sub syllogism \ --fixtures bench/fixtures/5s/syllogism-v1.jsonl -bench-5s-synthesis: bootstrap +bench-5s-synthesis: ## 5S Synthesis (derivation-pass-rate) $(PY) -m bench.batteries.runner --battery 5s --sub synthesis \ --fixtures bench/fixtures/5s/synthesis-v1.jsonl -bench-5s-semiotics: bootstrap +bench-5s-semiotics: ## 5S Semiotics (invariance-under-swap) $(PY) -m bench.batteries.runner --battery 5s --sub semiotics \ --fixtures bench/fixtures/5s/semiotics-v1.jsonl ``` -Existing `bench-5s` aggregate target picks them up via `--all` -once the default fixture map in `runner.py` includes them. - -### 4.4 Tests - -- Replace `test_stub_sub_batteries_return_zero` parametrizations - for the three sub-batteries with positive coverage: - `test_5s_syllogism_runs`, `test_5s_synthesis_runs`, - `test_5s_semiotics_runs`. -- Add fixture-digest stability test for each new fixture file. - -### 4.5 Bumps - -- New fixture files (additive). Phase 1a fixture digests stay - pinned. -- Runner-stub replacements in `b_5s.py`. -- New Makefile targets. -- No schema changes. No `governance_policy_hash` changes. +The existing `bench-5s` aggregate target picks them up via +`--all` once `_DEFAULT_FIXTURES` in `runner.py` includes them. --- -## 5. Out of scope +## 6. Tests -- 5T Phase 1b (Triangulation, Truthtables, Transitivity, Time — - ticket #000024). -- 5F battery (Function, Finetuning, Falsification, Formulate, - Feedback Loop — ticket #000025). -- 5R battery (workspace operators) — depends on SelfModel - (#000014, landed) + memory_root (#000017, landed); reserved as - Phase 2 of #000021. +Replace stub-zero parametrizations for the three sub-batteries. + +Required positive coverage: + +``` +test_5s_syllogism_runs +test_5s_synthesis_runs +test_5s_semiotics_runs +test_5s_syllogism_fixture_digest_stable +test_5s_synthesis_fixture_digest_stable +test_5s_semiotics_fixture_digest_stable +test_5s_phase1a_digest_unchanged # guard against regression +``` + +Modality-safety tests (carrier-schema future-readiness): + +``` +test_5s_fixture_accepts_carrier_metadata +test_5s_text_runner_rejects_unsupported_nontext_carrier_cleanly +test_5s_semiotics_synonym_swap_preserves_invariant_fields +test_5s_semiotics_bad_swap_fails +``` + +The last group enforces that the schema accepts carrier metadata +today even though only text is executed. + +--- + +## 7. Out of scope + +- Cross-domain π* composition for 5S (covered by ticket #000015, + already landed). +- Multimodal 5S execution (depends on #000013 spatial-temporal + substrate + π* image / audio / world canonicalizers; future + tickets). +- 5T Phase 1b (ticket #000024). +- 5F battery (ticket #000025). +- 5R battery — depends on SelfModel (#000014, landed) + + memory_root (#000017, landed); reserved as Phase 2 of #000021. - Cross-corpus expansion (Phase 3 of #000021). +- Hidden-channel **generation** or **concealment** tooling. + Detection-only is the boundary. --- -## 6. Risks & open questions +## 8. Risks & mitigations -- **Syllogism fixture quality.** Hand-curated chains can over-fit - to arborist's current claim-lattice parser. Mitigation: pull - from at least three different test files so coverage spans - multiple authoring styles. -- **Synthesis variance.** Depends on retrieval which has - per-shard variance. Mitigation: pin shard set in the fixture - meta; bench-maxing 5pp signal floor accounts for noise. -- **Semiotics blast radius.** A bad swap fixture could mask - genuine drift if the swap is too aggressive (e.g., swaps that - also change semantic meaning, not just surface). Mitigation: - curate from `concept_relations` of kind `synonym` only — those - are corpus-derived equivalences with known semantic preservation. +| Risk | Mitigation | +|---|---| +| Syllogism overfits to toy chains | Pull from ≥3 different test-source families; include invalid converses, missing-premise, unsupported-evidence-pointer failures. | +| Semiotics swaps accidentally change meaning | Use only `concept_relations.relation_kind='synonym'` for v1 positives; include explicit non-equivalent negative swaps. | +| Synthesis retrieval variance | Pin `shard_set` + `retrieval_mode` + `expected_fact_ids` in fixture meta; compare derivation shape, not exact prose. | +| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers rejected cleanly in v1; multimodal execution deferred to #000013 / #000015 follow-ups. | +| Steganography → dual-use tooling | v1 does not generate, hide, or recover covert messages. Future fixtures framed as defensive detection / flagging. | --- -## 7. Status +## 9. Closure criterion + +#000023 closes when: + +1. `run_syllogism`, `run_synthesis`, `run_semiotics` are all non-stub. +2. Each ships ≥30 deterministic fixtures with stable `fixture_digest`. +3. Fixtures include `carrier` / `domain` / `pi_star_ref` / + `loss_report_refs` / `modality_notes` fields. +4. Unsupported non-text carriers fail/skip explicitly with + `reason="unsupported_carrier"`. +5. `make bench-5s` runs all five 5S sub-batteries (Syntax, + Semantics, Syllogism, Synthesis, Semiotics). +6. Phase 1a fixture digests (`syntax-v1.jsonl`, + `semantics-v1.jsonl`) unchanged. +7. No schema, governance hash, or canonicalization-version changes. +8. Tests cover runner execution, fixture-digest stability, and + carrier-schema future-readiness. + +--- + +## 10. Status **Open · awaiting go/no-go.** Three sub-batteries × ~30 fixtures -× one runner each ≈ 600-800 lines of code + fixtures. Estimated -mid-sized session. +× one runner each ≈ 800-1000 lines code + fixtures. Mid-sized +session. -Closure criterion: all three runners non-stub, fixture files -landed with version pins, `make bench-5s` runs all five 5S -sub-batteries (Syntax, Semantics, Syllogism, Synthesis, Semiotics) -and emits a JSON result per sub-battery. Existing Phase 1a -fixtures keep their digests. +The architectural correction from the 2026-05-08 review: +**Semiotics, Syntax, and Semantics are defined over sign-bearing +carriers, not just written language.** Phase 1b ships the text +implementation; the fixture schema preserves the future path +into spatial-temporal world models, multimodal AGI evaluation, +and defensive hidden-channel detection. diff --git a/docs/tickets/ticket-000024-5t-phase-1b-and-rename.md b/docs/tickets/ticket-000024-5t-phase-1b-and-rename.md index 33114f3..add8c47 100644 --- a/docs/tickets/ticket-000024-5t-phase-1b-and-rename.md +++ b/docs/tickets/ticket-000024-5t-phase-1b-and-rename.md @@ -1,160 +1,318 @@ -# Ticket #000024 — 5T Phase 1b: complete sub-batteries + align vocabulary with Dav1DPrometheus +# Ticket #000024 — 5T Phase 1b: vocabulary alignment + carrier-aware completion **Status:** open · awaiting go/no-go **Opened:** 2026-05-07 -**Scope:** Complete the 5T battery — rename existing sub-batteries -to match Dav1DPrometheus's authoritative vocabulary -(Transfer → Transfer Learning, Truth → Truthtables, Timing → -Time), and ship real implementations + fixtures for the four -remaining stub sub-batteries (Triangulation, Truthtables, -Transitivity, Time). -**Audience:** fox + future bench authors + anyone composing a v8 -fitness target across the 5T axes. -**Hard constraint:** the rename is a NEW fixture-version landing -(`v1` → `v2`); existing `transfer-v1.jsonl` stays in place under -its sub-battery name `transfer` so #000021 Phase 1a digests don't -shift retroactively. New work uses the renamed identifiers. +**Revised:** 2026-05-08 (cross-modality correction folded in per +review response) +**Scope:** Two coupled jobs: +1. Align 5T vocabulary with Dav1DPrometheus's authoritative + formulation (Transfer → Transfer Learning, Truth → Truthtables, + Timing → Time). +2. Replace the four 5T zero-task stubs (Triangulation, Truthtables, + Transitivity, Time) with carrier-aware Phase-1b runners. +**Audience:** fox + future bench authors + downstream v8 / v7-W / +multimodal work needing fitness targets across 5T axes. +**Hard constraint:** Phase 1a `transfer-v1.jsonl` digest stays +pinned; new work uses `v2` fixture filenames and renamed sub-battery +identifiers. No schema bump, no governance hash bump, no +canonicalization-version bump. Unsupported non-text carriers MUST +fail or skip explicitly. No LLM-as-judge in any 5T runner. **Source:** Dav1DPrometheus (BasementAGI host) — *"5Ts: Transitivity, Transfer Learning, Triangulation, Truthtables, Time."* --- -## 1. Problem statement +## 1. Corrected purpose -Two coupled gaps: +**Vocabulary correction is not cosmetic.** Each rename pins a more +operationally precise concept: -### 1.1 Vocabulary drift from authoritative framework - -The SQD whitepaper used: **Transfer · Triangulate · Timing · -Transitivity · Truth.** - -Dav1DPrometheus's published framework uses: **Transitivity · -**Transfer Learning** · Triangulation · **Truthtables** · **Time**.** - -Three differences: - -| arborist (current) | Dav1DPrometheus | Why his is right | +| arborist Phase 1a | Dav1DPrometheus | Why his is right | |---|---|---| -| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; "Transfer Learning" pins the concrete ML primitive. | -| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the concrete logical-scenario coverage primitive. | -| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, which is the actual semantic. | +| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; pins the concrete ML primitive (cross-task / cross-domain method preservation). | +| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the explicit logical-state coverage primitive. | +| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, lineage, memory, stale-state handling. | -arborist landed Phase 1a with the SQD-whitepaper names. Per memory -note `dav1dprometheus_framework.md`, his wording is authoritative — -arborist honors it. +The architectural correction (folded in 2026-05-08): -### 1.2 Four stub sub-batteries return zero tasks +> **5T must not be permanently language-only or prose-only. +> Transitivity, Transfer Learning, Triangulation, Truthtables, and +> Time must eventually operate across text, code, tables, images, +> audio, sensor streams, spatial state, world-model state, and +> hidden or alternate media carriers.** -Same as 5S (ticket #000023): Triangulation, Truthtables, -Transitivity, Time all currently return zero-task `BatteryResult` -objects via `_stub_result` in `bench/batteries/b_5t.py`. Without -real implementations: - -| Sub-battery | Failure mode under stubs | -|---|---| -| **Triangulation** | No bench for "do independent verifiers agree?" — directly the v7 §11.4 cross-source consistency claim. | -| **Truthtables** | No coverage of "for these N propositional variables, does the system handle all 2^N cases?" — gates v8 selection's ability to certify logical-completeness claims. | -| **Transitivity** | No bench for "if A→B and B→C, does the system derive A→C?" — multi-step entailment is a SelfModel capability claim with no current measurement. | -| **Time** | No bench for "does the system maintain temporal context across queries?" — directly relates to memory_root + SelfModel's lineage chain, both shipped but unbenchmarked. | +Phase 1b implements **text / claim-lattice / memory-root** carriers +only. The fixture schema is carrier-aware so future phases can +attach code / image / audio / sensor / world canonicalizers via +the π* registry without re-authoring fixtures. --- -## 2. Design choices +## 2. Carrier-aware 5T definitions -### 2.1 Rename strategy +| Sub-battery | Definition (carrier-general) | Phase 1b carrier | +|---|---|---| +| **Transfer Learning** | Whether a learned structure or method transfers across a task, domain, or carrier. Future: text rule → table task; diagram relation → textual derivation; world-state pattern → claim-lattice conclusion. | text / claim_lattice | +| **Triangulation** | Independent evidence, verifier, or modality paths converging on the same claim state. Future: text + table + chart agreement; image/OCR + caption + metadata; sensor + event log + textual report. | claim_lattice (existing four verifier strategies) | +| **Transitivity** | Relation chains preserving valid inference across steps. Future: spatial + visual relations; code dependency + package metadata; sensor temporal-ordering. | claim_lattice (typed relation graph) | +| **Truthtables** | Exhaustive logical-state coverage. Future: visual state toggles; world-state propositions; sensor Boolean predicates; tool precondition matrices. | propositional_logic (N=2..4 variables) | +| **Time** | Temporal-context maintenance, lineage, memory, stale-state handling. Future: video frames; sensor streams; world-state transitions; tool/action DAG timelines. | memory_snapshot / selfmodel_snapshot | -**A. Hard rename in place.** Update `b_5t.py` function names and -fixture filenames; existing Phase 1a digests change. Cleanest -forward but invalidates Phase 1a results. - -**B. Coexist v1 + v2.** Keep `transfer-v1.jsonl` + `run_transfer` -function in place; add `transfer-learning-v2.jsonl` + -`run_transfer_learning`. New work uses v2 names. Old `bench-5t` -target still runs against v1. - -**C. Soft rename (alias).** Keep file names; add alias mapping in -the runner so `--sub transfer-learning` resolves to `run_transfer`. -Code stays ugly forever. - -**Recommendation:** B. Phase 1a was honest at the time it landed -(SQD whitepaper was the only source then); we don't retroactively -break it. New work explicitly uses Dav1DPrometheus's vocabulary. - -### 2.2 Fixture sourcing - -- **Triangulation** — replay arborist's existing four-strategy - verifier (quote/span/entity/paraphrase) on a fixture set; - metric is the per-task agreement rate. Already-shipped verifier - surface; fixtures are easy. -- **Truthtables** — small-N propositional fixtures (N=2..4); each - fixture lists all 2^N input combinations + the expected output; - arborist's claim-lattice has to evaluate every row. New surface - (currently arborist doesn't do propositional eval); requires a - thin wrapper. -- **Transitivity** — multi-step claim chains pulled from existing - claim-lattice tests; metric is full-chain-pass-rate. -- **Time** — multi-query sequences against the same shard set; - metric is "does memory_root snapshot N+1 preserve information - from snapshot N?" Direct integration with memory-root (#000017, - landed). - -### 2.3 Cross-reference to memory-root and SelfModel - -The Time sub-battery implementation reads memory_records and -selfmodel_records to assert temporal-context preservation. This -makes #000024 the first ticket where the v8 substrate -(SelfModel + memory_root) gets a measurable bench target, not -just an audit-chain artifact. +Time is the **first sub-battery where v8 substrate components** +(SelfModel #000014 + memory_root #000017) become measurable bench +targets. That is structurally significant: until #000024, those +substrates were audit-chain artifacts only; now they have a +fitness signal. --- -## 3. Recommendation +## 3. Fixture schema (carrier-aware) -Two-phase landing: +```json +{ + "id": "5t--NNN", + "battery": "5t", + "sub_battery": "time", + "version": "v1", -**Phase 1b.1 — Rename + cosmetic alignment.** Add v2 fixture file -names + new sub-battery aliases. Keep v1 in place. ~100 lines. + "carrier": "memory_snapshot", + "domain": "memory_root", + "pi_star_ref": "pi_memory_v1", + "loss_report_refs": [], + "modality_notes": "Phase 1b: text/claim-lattice/memory-root only; future carriers may include code/table/image/audio/sensor/world.", -**Phase 1b.2 — Implement four sub-batteries.** Triangulation -first (existing surface), then Transitivity (lexical), then -Truthtables (new wrapper), then Time (memory-root-integrated). -~600-800 lines. - ---- - -## 4. Implementation sketch - -### 4.1 Fixture files (new) - -``` -bench/fixtures/5t/transfer-learning-v2.jsonl # rename of transfer-v1 -bench/fixtures/5t/triangulation-v1.jsonl -bench/fixtures/5t/truthtables-v1.jsonl -bench/fixtures/5t/transitivity-v1.jsonl -bench/fixtures/5t/time-v1.jsonl + "task": { ... } +} ``` -`transfer-v1.jsonl` stays in place; `bench-5t-transfer` still runs -it. Phase 1a tests stay green. +Phase 1b allowed carriers: `text`, `claim_lattice`, +`memory_snapshot`, `selfmodel_snapshot`, `verifier_strategies`, +`propositional_logic`, `relation_graph`. Other values → +`reason="unsupported_carrier"`. -### 4.2 Runner additions +--- -In `bench/batteries/b_5t.py`, real implementations replace stubs: +## 4. Per-sub-battery design -- `run_transfer_learning(fixtures_path)` — same logic as - `run_transfer`; alias. -- `run_triangulation(fixtures_path)` — for each task, call - arborist's four verifier strategies on (claim, evidence) pairs - and report agreement rate. -- `run_truthtables(fixtures_path)` — for each task, evaluate the - claim-lattice over 2^N input combinations. -- `run_transitivity(fixtures_path)` — chase A→B, B→C through the - claim-lattice; assert C derives. -- `run_time(fixtures_path)` — multi-query memory_root preservation; - reads `memory_records` to assert continuity. +### 4.1 Transfer Learning (rename of Phase 1a `transfer`) -### 4.3 Makefile additions +**Metric:** `transfer_learning_success_rate`. + +**Backward compat:** Keep `transfer-v1.jsonl`, `run_transfer`, +`bench-5t-transfer` in place — Phase 1a digest stays pinned. Add +`transfer-learning-v2.jsonl`, `run_transfer_learning`, +`bench-5t-transfer-learning` for new work. + +**Fixture shape:** + +```json +{ + "id": "5t-transfer-learning-001", + "version": "v2", + "carrier": "text", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "source_task": { + "pattern": "A supports B when cited span contains anchor B.", + "example": "Release-date claim with date anchor." + }, + "target_task": { + "pattern": "Founder-date claim with date anchor.", + "expected_transfer": true + }, + "expected": "pass" +} +``` + +### 4.2 Triangulation + +**Metric:** `triangulation_agreement_rate`, +`strategy_disagreement_rate`. + +**Fixture shape:** + +```json +{ + "id": "5t-triangulation-001", + "carrier": "claim_lattice", + "domain": "verifier_strategies", + "pi_star_ref": "claim-lattice@v1", + "claim": "Back to the Future was released on July 3, 1985.", + "evidence": ["E1", "E2"], + "strategies": ["quote", "span", "entity", "paraphrase"], + "expected_agreement_min": 0.75 +} +``` + +**Runner:** invokes arborist's existing four verifier strategies +on (claim, evidence) pairs, computes pairwise agreement, passes +when agreement ≥ `expected_agreement_min`. + +**Future-modality hook:** + +```json +{ + "future_modality_paths": [ + {"carrier": "table", "role": "structured evidence"}, + {"carrier": "image", "role": "visual confirmation"} + ] +} +``` + +v1 runner ignores; fixture is forward-compatible. + +### 4.3 Transitivity + +**Metric:** `full_chain_pass_rate`. + +**Critical correction:** not all relations are transitive. Phase +1b ships a typed-relation-whitelist: + +``` +implies +subset_of +ancestor_of +before +less_than +``` + +Runners reject mixed-relation chains and non-transitive relations +(`related_to`, `causes` without domain constraints, etc.). + +**Positive fixture:** + +```json +{ + "id": "5t-transitivity-001", + "carrier": "claim_lattice", + "domain": "relation_graph", + "pi_star_ref": "pi_relation_graph_v1", + "edges": [ + {"from": "A", "to": "B", "relation": "implies"}, + {"from": "B", "to": "C", "relation": "implies"} + ], + "query": {"from": "A", "to": "C", "relation": "implies"}, + "expected": "pass" +} +``` + +**Negative fixture:** + +```json +{ + "id": "5t-transitivity-NEG-001", + "edges": [ + {"from": "A", "to": "B", "relation": "related_to"}, + {"from": "B", "to": "C", "relation": "causes"} + ], + "query": {"from": "A", "to": "C", "relation": "causes"}, + "expected": "fail", + "reason": "mixed relation types do not license transitive closure" +} +``` + +### 4.4 Truthtables + +**Metric:** `truth_table_coverage_rate`, `row_accuracy_rate`. + +**Constraint:** N=2..4 variables only (avoids combinatorial +blowup; keeps logical coverage real). + +**Fixture shape:** + +```json +{ + "id": "5t-truthtables-001", + "carrier": "claim_lattice", + "domain": "propositional_logic", + "pi_star_ref": "pi_truth_table_v1", + "variables": ["A", "B"], + "expression": "A AND B", + "rows": [ + {"inputs": {"A": false, "B": false}, "expected": false}, + {"inputs": {"A": false, "B": true}, "expected": false}, + {"inputs": {"A": true, "B": false}, "expected": false}, + {"inputs": {"A": true, "B": true}, "expected": true} + ] +} +``` + +**Runner:** parse expression → enumerate rows → evaluate each +deterministically → fixture fails if any row mismatches. +Deterministic parser; no LLM. + +### 4.5 Time + +**Metric:** `temporal_context_preservation_rate`, +`stale_marking_rate`, `memory_lineage_pass_rate`. + +**Fixture shape:** + +```json +{ + "id": "5t-time-001", + "carrier": "memory_snapshot", + "domain": "memory_root", + "pi_star_ref": "pi_memory_v1", + "snapshots": [ + {"snapshot_id": "S0", "memory_root": "M0", "facts": ["X is true as of t0"]}, + {"snapshot_id": "S1", "memory_root": "M1", "facts": ["X is revised as of t1"]} + ], + "expected": { + "preserves_prior": true, + "marks_stale": ["X is true as of t0"], + "current_root": "M1" + }, + "policy_assumption": { + "memory_root_binding": "advisory", + "selfmodel_integration": true + } +} +``` + +**Runner:** reads `memory_records` (and `selfmodel_records` where +needed) → compares snapshot N and N+1 → asserts temporal +preservation, stale marking, lineage continuity. + +**Future-time-carriers (documented, not executed in v1):** + +```json +{ + "future_time_carriers": [ + {"carrier": "video", "domain": "frame_sequence"}, + {"carrier": "sensor", "domain": "time_series"}, + {"carrier": "world", "domain": "state_transition_graph"} + ] +} +``` + +--- + +## 5. Required file changes + +### 5.1 Fixture files + +``` +bench/fixtures/5t/transfer-learning-v2.jsonl # new ≥30 tasks +bench/fixtures/5t/triangulation-v1.jsonl # new ≥30 tasks +bench/fixtures/5t/truthtables-v1.jsonl # new ≥30 tasks (N=2..4) +bench/fixtures/5t/transitivity-v1.jsonl # new ≥30 tasks (whitelist) +bench/fixtures/5t/time-v1.jsonl # new ≥30 tasks (memory) + +bench/fixtures/5t/transfer-v1.jsonl # UNCHANGED (Phase 1a digest pinned) +``` + +### 5.2 Runner functions + +In `bench/batteries/b_5t.py`: + +- Keep `run_transfer` (Phase 1a) intact. +- Add `run_transfer_learning`, `run_triangulation`, + `run_truthtables`, `run_transitivity`, `run_time` as non-stub + implementations. + +### 5.3 Make targets ```makefile bench-5t-transfer-learning: bootstrap @@ -178,61 +336,100 @@ bench-5t-time: bootstrap --fixtures bench/fixtures/5t/time-v1.jsonl ``` -Existing `bench-5t` aggregate keeps `transfer` (v1) for back-compat. - -### 4.4 Tests - -- Add `test_5t_transfer_learning_alias` — confirms new name resolves - to same surface as `transfer`. -- Add positive tests for each new sub-battery. -- Update stub-test parametrization to remove the four sub-batteries - that get real implementations. - -### 4.5 Bumps - -- New fixture files (additive). -- New runner functions; old ones (`run_transfer`, - `run_triangulate`, etc.) stay in place for back-compat. -- New Makefile targets. -- No schema changes. +`bench-5t` aggregate keeps `transfer` (v1) for backward +compatibility. New canonical aggregate is `bench-5t-v2` covering +the five Dav1DPrometheus sub-batteries; or operators wire +`bench-5t` to the v2 names once the rename has stabilized. --- -## 5. Out of scope +## 6. Tests -- 5S Phase 1b — ticket #000023. -- 5F battery (entirely new) — ticket #000025. -- 5R battery (workspace operators) — Phase 2 of #000021. -- Retiring the v1 names. Once v2 is the standard, deprecate v1 - in a follow-up ticket; this ticket leaves both live. +``` +test_5t_transfer_learning_runs +test_5t_triangulation_runs +test_5t_truthtables_runs +test_5t_transitivity_runs +test_5t_time_runs +test_5t_v1_transfer_digest_unchanged # guard regression +test_5t_fixture_accepts_carrier_metadata +test_5t_text_runner_rejects_unsupported_nontext_carrier_cleanly +test_5t_time_reads_memory_root_fixture +test_5t_transitivity_rejects_non_whitelisted_relation +test_5t_truthtables_N_le_4_only +``` + +Stable-digest tests for each new fixture file. Remove the four +implemented sub-batteries from any "stub returns zero" +parametrization. --- -## 6. Risks & open questions +## 7. Out of scope -- **Truthtables surface.** arborist doesn't currently do - propositional evaluation; building the wrapper is real work. - Defer if scope compresses; ship the other three first. -- **Time integration depth.** memory_root snapshots are advisory - (default policy off). The Time bench can still measure - preservation across snapshots even when binding is off, but - results may not reflect production behavior under different - policies. Document the policy assumption in the fixture meta. -- **Backward compatibility.** Old `bench-5t` (v1 names) stays - alongside new `bench-5t-*` (v2 names). Risk of confusion; - mitigate by clear naming + a `docs/bench-vocabulary.md` note - if needed. +- Multimodal 5T execution (depends on #000013 spatial-temporal + substrate + π* image / audio / world canonicalizers). +- 5S Phase 1b (ticket #000023). +- 5F battery (ticket #000025). +- 5R battery — Phase 2 of #000021. +- Retiring v1 names. Once v2 is the standard, deprecate v1 in a + follow-up; this ticket leaves both live. +- Hidden-channel **generation**. Defensive detection only. --- -## 7. Status +## 8. Risks & mitigations -**Open · awaiting go/no-go.** Mid-to-large surface; biggest sub- -battery (Time) needs memory-root integration. Recommended -implementation sequence: rename first, then sub-batteries in the -order Triangulation → Transitivity → Time → Truthtables. +| Risk | Mitigation | +|---|---| +| Vocabulary confusion (v1 vs v2) | Keep v1 transfer fixtures; add v2 transfer-learning fixtures; document Dav1DPrometheus canonical vocabulary. | +| Truthtables scope blowup | N=2..4 only; deterministic parser; no LLM-as-judge. | +| Transitivity overclaims | Typed relation graph; transitive-relation whitelist; negative fixtures for non-transitive relation types. | +| Time policy ambiguity | Fixture meta includes `memory_root_binding` policy assumption; runner reports policy mismatch. | +| Premature multimodal scope creep | Carrier metadata now; non-text carrier support later; unsupported carriers fail/skip explicitly. | -Closure criterion: all five 5T sub-batteries (Transfer Learning, -Triangulation, Truthtables, Transitivity, Time) have non-stub -runners + fixture files; `make bench-5t` runs the full set under -v2 names; v1 `bench-5t-transfer` still passes. +--- + +## 9. Closure criterion + +#000024 closes when: + +1. Dav1DPrometheus vocabulary present: Transfer Learning, + Triangulation, Truthtables, Transitivity, Time. +2. Legacy `transfer-v1.jsonl` digest unchanged. +3. New `transfer-learning-v2.jsonl` lands. +4. Triangulation runner non-stub. +5. Transitivity runner non-stub with typed-relation whitelist. +6. Truthtables runner non-stub with N=2..4 cap. +7. Time runner non-stub and reads `memory_records` / + `selfmodel_records` surfaces. +8. All new fixtures include `carrier` / `domain` / `pi_star_ref` / + `loss_report_refs` / `modality_notes`. +9. Unsupported future carriers fail or skip explicitly. +10. Make targets for each sub-battery pass. +11. Phase 1a fixture digests unchanged. +12. No schema, governance hash, or canonicalization-version changes. + +--- + +## 10. Status + +**Open · awaiting go/no-go.** Two-phase landing recommended: + +- **Phase 1b.1 — vocabulary alignment.** Add v2 fixture file names + + new sub-battery aliases. Keep v1 in place. ~150 lines. +- **Phase 1b.2 — implement four sub-batteries.** Triangulation + first (existing surface), Transitivity second (typed-relation + whitelist), Time third (memory-root integrated), Truthtables + last (new propositional wrapper). ~700-900 lines + fixtures. + +The architectural correction: + +> **5T evaluates reasoning transformations across states, domains, +> evidence paths, logical cases, and time. Those transformations +> can live in text today, but must be representable across any +> sign-bearing or state-bearing carrier later.** + +This makes #000024 the first ticket where v8 substrate becomes +measurable bench territory and the path to spatial-temporal / +multimodal / world-model AGI evaluation stays open. diff --git a/docs/tickets/ticket-000025-5f-battery.md b/docs/tickets/ticket-000025-5f-battery.md index 59192aa..a9f434f 100644 --- a/docs/tickets/ticket-000025-5f-battery.md +++ b/docs/tickets/ticket-000025-5f-battery.md @@ -1,190 +1,376 @@ -# Ticket #000025 — 5F battery: Function, Finetuning, Falsification, Formulate, Feedback Loop +# Ticket #000025 — 5F battery: Function · Finetuning · Falsification · Formulate · Feedback Loop **Status:** open · awaiting go/no-go **Opened:** 2026-05-07 -**Scope:** Open the 5F battery — Dav1DPrometheus's third evaluation -axis, missing entirely from arborist before this ticket. Each sub- -battery measures a distinct dimension of agent quality: -**Function** (task alignment), **Finetuning** (adaptation -efficiency), **Falsification** (error detection + correction), -**Formulate** (logical structuring), **Feedback Loop** (continuous -improvement). -**Audience:** fox + future bench authors. Once landed, 5F + 5S + 5T -together form the complete non-embodied AGI evaluation surface +**Revised:** 2026-05-08 (cross-modality + state-space synthesis +folded in per review response) +**Scope:** Open the 5F battery — Dav1DPrometheus's +operational-quality axis, missing entirely from arborist before +this ticket. Five sub-batteries: **Function** (task alignment), +**Finetuning** (adaptation efficiency), **Falsification** (error +detection + correction), **Formulate** (logical structuring), +**Feedback Loop** (continuous improvement). 5F integrates with +existing arborist surfaces — `providence_cache.falsification_state`, +`audit_events`, `memory_records` (#000017, landed), +`selfmodel_records` (#000014, landed). No new substrate work. +**Audience:** fox + future bench authors. Once 5F lands, 5S + 5T + +5F together form the complete non-embodied AGI evaluation surface Dav1DPrometheus published. -**Hard constraint:** 5F integrates with surfaces that already exist -in arborist — `providence_cache.falsification_state`, `audit_events`, -`memory_records` (#000017), `selfmodel_records` (#000014). Don't -require new substrate work to land 5F. +**Hard constraint:** 5F is text-first in Phase 1a but the fixture +schema is **carrier-aware**. Unsupported non-text carriers MUST +fail or skip explicitly. No LLM-as-judge in any 5F runner. No +schema bump, no governance hash bump. Hidden-channel work stays +defensive (detection only, never generation). **Source:** Dav1DPrometheus (BasementAGI host) — *"5Fs: Function, -Finetuning, Falsification, Formulate, Feedback Loop."* Honoring his -legacy. +Finetuning, Falsification, Formulate, Feedback Loop."* --- -## 1. Problem statement +## 1. Why 5F is the operational bridge -Dav1DPrometheus's framework names three orthogonal evaluation axes: - -```text -5Ss = linguistic precision (syntax + semantics + syllogism + synthesis + semiotics) -5Ts = temporal/cross reasoning (transitivity + transfer learning + triangulation + truthtables + time) -5Fs = operational quality (function + finetuning + falsification + formulate + feedback loop) -``` - -arborist's bench harness (ticket #000021) implements 5S + 5T as -fixture-driven batteries. **5F has no representation in arborist.** -The original SQD whitepaper omitted it; Dav1DPrometheus's framework -is the only source for it. +5S evaluates linguistic / sign-structure precision. +5T evaluates temporal / cross-reasoning / logical coverage. +**5F evaluates whether the organism acts, adapts, falsifies, +formulates, and integrates feedback** — the dimensions that turn a +state-space into an *adaptive* state-space. Without 5F: -| Sub-battery | Why arborist needs it | -|---|---| -| **Function** | Measures whether the system actually executes its designated task. arborist's QA layer has answer-mode bench coverage but not a clean "did the system DO what it was asked" metric. | -| **Finetuning** | Measures speed/efficiency of adaptation. SelfModel parent→child transitions are unbenchmarked. | -| **Falsification** | Error-detection accuracy is core to v9.8 admissibility, but `providence_cache.falsification_state` transitions are not benched. | -| **Formulate** | Coherence of organized knowledge structures — directly the claim-lattice's job, currently only measured by per-claim verification rate, not chain-level coherence. | -| **Feedback Loop** | Integration efficiency for incoming corrections. Memory-root branch projections (#000017) update on snapshot but speed of integration is unmeasured. | - -### 1.1 Why this is a separate ticket from #000021 - -#000021 was scoped to the SQD whitepaper's 5S + 5T. Adding 5F -would have changed scope mid-design. Treating 5F as its own -ticket keeps each battery's story coherent and lets fox approve -or punt 5F independently. - ---- - -## 2. Sub-battery semantics (per Dav1DPrometheus) - -### 2.1 Function - -> *Task Alignment: Precision in executing designated tasks. -> Operational Effectiveness: Empirical success rate in task execution.* - -Each task: a directive (e.g., "summarize document X"), a target -output shape (e.g., claim-lattice with N claims), and an -evaluation predicate (does the produced output meet the shape?). -Metric: function-pass-rate. - -Maps to arborist surfaces: `arborist ask` invocations under -specific `answer_mode` settings. - -### 2.2 Finetuning - -> *Performance Improvement: Quantitative percentage increase in -> performance metrics post-finetuning. -> Adaptability: Efficiency in adapting to new tasks, measured by -> time and resources required.* - -Each task: a parent SelfModel + a target capability claim that -the parent fails + a child SelfModel that should pass. Metric: -fraction of parent→child transitions where the child's measured -metric improves over parent's. - -Maps to arborist surfaces: `selfmodel_records` parent→child chain -+ `selfmodel_capability_claims` measured_value tracking. - -### 2.3 Falsification - -> *Error Detection Accuracy: Rate of correctly identifying system -> errors. Correction Effectiveness: Success rate in rectifying -> identified errors.* - -Each task: a known-bad providence record (planted) + the expected -falsification reason. Metric: fraction of planted records the -verifier correctly flags + fraction of repair operations that -restore correctness. - -Maps to arborist surfaces: `falsification_state` transitions, -`providence_repair` audit events, `arborist.qa.repair`. - -### 2.4 Formulate - -> *Logical Structuring: Degree of coherence in organized -> knowledge structures. -> Framework Clarity: Ease of comprehension and interpretability of -> structured knowledge.* - -Each task: a free-text input + the expected claim-lattice -structure (N claims, ordered, with pointer IDs). Metric: structural- -match-rate (sorted-claim-text + pointer-ID-set agreement, NOT -exact-string). - -Maps to arborist surfaces: `arborist.qa.parse_claims` + -`arborist.pi_star.claim_lattice`. - -### 2.5 Feedback Loop - -> *Integration Efficiency: Speed and efficacy in assimilating -> feedback into system processes. -> Continuous Improvement: Measurable performance enhancements -> through iterative feedback mechanisms.* - -Each task: a sequence of (operation, observation) pairs forming a -feedback chain; the system should integrate observations into -subsequent operations. Metric: integration-coverage-rate (fraction -of observations that affect a downstream snapshot). - -Maps to arborist surfaces: `memory_branch_summaries` updates + -`audit_events` chain delta between snapshots. - ---- - -## 3. Design choices - -### 3.1 Battery vs sibling layer - -**A. Treat 5F as a peer to 5S/5T inside `bench/batteries/`.** Same -runner skeleton; new file `b_5f.py`. - -**B. Treat 5F as a separate top-level harness.** New module -`bench/quality/`. Distinct because Function/Finetuning are -operational rather than substrate-evaluation properties. - -**Recommendation:** A. Keeps the harness uniform; the runner CLI -already supports arbitrary battery names. The semantic distinction -between 5S/5T (substrate) and 5F (operational) is documented in -the ticket but doesn't need a code-level distinction. - -### 3.2 Fixture sourcing per sub-battery - -| Sub-battery | Fixture source | -|---|---| -| Function | Hand-curated `(directive, expected-shape, evaluator)` triples. | -| Finetuning | Generated from `selfmodel_records` chains in actual shards (or test fixtures). | -| Falsification | Planted-error fixtures: known-bad providence records with stated falsification reason. | -| Formulate | Hand-curated `(text, expected-claim-lattice)` pairs from existing claim-lattice tests. | -| Feedback Loop | Multi-snapshot fixture chains; each chain tagged with expected memory-root delta semantics. | - -### 3.3 Phase staging - -Like 5S/5T, 5F lands in two phases: - -- **Phase 1a (this ticket scope):** runner skeleton in `b_5f.py`, - ~10 seed fixtures per sub-battery, all sub-batteries return real - results (no stubs). Get the surface up. -- **Phase 1b (follow-up):** expand fixtures to 30-50 per sub-battery - for bench-floor signal. - ---- - -## 4. Implementation sketch - -### 4.1 Module + fixtures - ``` -bench/batteries/b_5f.py # new; Function/Finetuning/Falsification/Formulate/Feedback runners -bench/fixtures/5f/function-v1.jsonl -bench/fixtures/5f/finetuning-v1.jsonl -bench/fixtures/5f/falsification-v1.jsonl -bench/fixtures/5f/formulate-v1.jsonl -bench/fixtures/5f/feedback-loop-v1.jsonl +state-space exists but may not improve. ``` -### 4.2 Runner registration +With 5F: + +``` +state-space becomes an adaptive learning organism. +``` + +That is the operational gap 5F closes. + +--- + +## 2. State-space / time synthesis + +The folded-in correction (2026-05-08 review): + +> **SQD + Merkle-AGI v7 + 5S/5T/5F + arborist together instantiate +> an explicit discrete state-space/time in which counting, +> mathematics, and logic become auditable operations over committed +> state — not just text generated by a latent model.** + +Define an arborist organism state at time `t`: + +``` +Ω_t = ( + W_t, workspace state + I_t, invariant objects under π* + C_t, concept / semiotic state + L_t, claim lattice + MRoot_t, memory root + SMRoot_t, SelfModel root + PRoot_t, policy root + BRoot_t, benchmark root + ARoot_t, audit root +) +``` + +A transition is `Ω_{t+1} = T(Ω_t, Δ_t)` where `Δ_t ∈ {new evidence, +new claim, new falsification, new benchmark result, new SelfModel +update, new memory snapshot, new feedback observation, new repair +operation}`. This is a discrete state-space/time — not metaphorical. +The system has states, transitions, invariants, events, memory, +verification, branching, rollback, selection pressure. + +5F maps onto the transitions: + +| Sub-battery | Transition question | +|---|---| +| Function | Does the transition do the intended operation? | +| Finetuning | Does the organism improve across parent→child states? | +| Falsification | Does the organism reject invalid states? | +| Formulate | Does the organism convert raw input into structured state? | +| Feedback Loop | Does observation at `t` affect state at `t+1`? | + +**Counting** emerges over finite committed sets (claims, verifier +passes, memory branches, feedback observations integrated, SelfModel +capability upgrades). **Mathematics** emerges over π*-canonical +invariants. **Logic** emerges from verifier predicates, truthtables, +syllogisms, transitivity, falsification states, admissibility +labels. **Time** emerges from ordered state transitions across +MemoryRoot / SelfModel / BenchmarkRoot. + +The break from traditional LLM stacks: + +> **Traditional LLMs generate math/logic as behavior. This stack +> externalizes math/logic as committed, verifier-addressable state.** + +That is the architectural justification for 5F's existence as a +distinct axis. + +--- + +## 3. Carrier-aware 5F definitions + +Common rule: + +> A 5F task evaluates whether the organism can operate correctly +> over a carrier, adapt over that carrier, falsify errors in that +> carrier, formulate structure from it, or integrate feedback about +> it. + +Phase 1a allowed carriers: + +``` +text · claim_lattice · providence_record · +selfmodel_snapshot · memory_snapshot · audit_event +``` + +Future carriers: + +``` +code · table · image · audio · video · sensor · +spatial/world states · tool_action · mixed · hidden_channel +``` + +Unsupported carriers in v1 → `reason="unsupported_carrier"`, +explicit fail or skip. + +--- + +## 4. Fixture schema + +```json +{ + "id": "5f--NNN", + "battery": "5f", + "sub_battery": "function", + "version": "v1", + + "carrier": "claim_lattice", + "domain": "qa_answer", + "pi_star_ref": "claim-lattice@v1", + "loss_report_refs": [], + "modality_notes": "Phase 1a uses text/claim-lattice; future carriers may include image, code, tool traces, world state.", + + "task": { ... } +} +``` + +--- + +## 5. Per-sub-battery design + +### 5.1 Function + +**Definition:** does the system actually perform the designated +operation? Phase 1a: directive → expected output shape → +deterministic predicate. + +**Fixture shape:** + +```json +{ + "id": "5f-function-001", + "carrier": "claim_lattice", + "domain": "qa_answer", + "pi_star_ref": "claim-lattice@v1", + "directive": "Return exactly two claim-lattice claims with evidence pointers.", + "answer_mode": "claim_lattice", + "expected_shape": { + "claim_count": 2, + "pointers_required": true + }, + "evaluator": "shape_match", + "expected": "pass" +} +``` + +**v1 evaluators (deterministic only):** + +``` +shape_match · pointer_set_match · threshold_on_metric +``` + +**Future evaluators:** `ast_patch_applies`, `image_region_detected`, +`world_state_updated`, `tool_postcondition_met`. + +**Metric:** `function_pass_rate = passed / total`. + +### 5.2 Finetuning + +**Definition:** measured improvement from parent state to child +state after adaptation. Adaptation includes SelfModel update, policy +update, benchmark-selected patch, π* canonicalizer update, +memory-root update, model-profile change, actual model finetune. + +**Fixture shape:** + +```json +{ + "id": "5f-finetuning-001", + "carrier": "selfmodel_snapshot", + "domain": "capability_transition", + "pi_star_ref": "pi_selfmodel_v1", + "parent_selfmodel": "SM_PARENT", + "child_selfmodel": "SM_CHILD", + "target_capability": "CAP-5S-SYLLOGISM", + "parent_measured_value": 0.45, + "child_measured_value": 0.65, + "expected_improvement_min": 0.05, + "resource_budget": { + "max_compute_ms_delta": 1000, + "max_storage_delta_bytes": 1000000 + }, + "expected": "pass" +} +``` + +**Metrics:** + +``` +adaptation_improvement_rate +adaptation_efficiency = improvement_delta / capital_cost_delta +``` + +`adaptation_efficiency` hooks into the capital ledger (#000020, +landed) and prepares fitness-cost weighing for v8 fork choice +(#000012). + +### 5.3 Falsification + +**Definition:** does the system detect and correct errors? + +**Fixture shape:** + +```json +{ + "id": "5f-falsification-001", + "carrier": "providence_record", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "record": "planted_bad_providence_record", + "expected_reason": "WARRANT_MISSING", + "verifier_method_root": "...", + "repair_expected": true, + "expected": "pass" +} +``` + +**Metrics:** `error_detection_rate`, `repair_success_rate`. + +**Verifier-method-root tag:** every Falsification fixture pins the +`verifier_method_root` it was authored against. If the verifier +shape changes upstream, the bench reports a *mismatch warning* +instead of treating the fixture as a false failure. + +**Hidden / alternate-media safety framing:** future carriers +(`carrier: "image"` / `"audio"` / `"hidden_channel"`) enable +defensive Falsification tasks — *detect unexpected carrier*, +*flag projection-loss risk*, *flag hidden-channel suspicion*, +*reject unsupported semantics*. Never generation, never +concealment. + +### 5.4 Formulate + +**Definition:** turn unstructured input into coherent, interpretable +structure. + +Phase 1a: free text → claim lattice. + +Future: image → scene graph; table → relational model; code → +AST/dependency-graph; sensor stream → event graph; world state → +object-relation graph; mixed media → cross-modal claim lattice. + +**Fixture shape:** + +```json +{ + "id": "5f-formulate-001", + "carrier": "text", + "domain": "claim_lattice", + "pi_star_ref": "claim-lattice@v1", + "input_text": "Document sentence with two claims and two pointers.", + "expected_lattice": { + "claim_count": 2, + "ordered": true, + "pointer_id_sets": [["E1"], ["E2"]] + }, + "match_policy": { + "claim_text": "sorted_approx", + "pointer_ids": "exact_set", + "order": "required" + }, + "expected": "pass" +} +``` + +**Metric:** `structural_match_rate`. + +**Match policy:** structural (sorted-approx claim text + +exact-set pointer IDs + required order), NOT exact-string. Avoids +brittleness without sacrificing determinism. + +### 5.5 Feedback Loop + +**Definition:** do observations at `t` change downstream +behavior/state at `t+1`? + +This is the most important 5F component for recursive +self-improvement. Without Feedback Loop, the state-space exists +but does not learn. + +**Fixture shape:** + +```json +{ + "id": "5f-feedback-loop-001", + "carrier": "memory_snapshot", + "domain": "memory_root", + "pi_star_ref": "pi_memory_v1", + "chain": [ + {"operation": "snapshot", "observation": "claim X is stale"}, + {"operation": "update_memory", "observation": "memory branch marks X stale"}, + {"operation": "snapshot", "expected_delta": "X appears in stale set"} + ], + "expected": "pass" +} +``` + +**Metrics:** + +``` +integration_coverage_rate +feedback_latency +downstream_effect_rate +feedback_efficiency = downstream_effect_count / capital_cost_delta +``` + +`feedback_efficiency` again hooks the capital ledger (#000020). + +**Phase 1a uses existing surfaces only:** `memory_records`, +`memory_branch_summaries`, `audit_events`. No new substrate work. + +--- + +## 6. Required file changes + +### 6.1 Module + fixtures + +``` +bench/batteries/b_5f.py # new +bench/fixtures/5f/function-v1.jsonl # ~10 seed (Phase 1a) +bench/fixtures/5f/finetuning-v1.jsonl # ~10 seed +bench/fixtures/5f/falsification-v1.jsonl # ~10 seed +bench/fixtures/5f/formulate-v1.jsonl # ~10 seed +bench/fixtures/5f/feedback-loop-v1.jsonl # ~10 seed +``` + +Each fixture file carries the standard `_meta` head with +`battery="5f"`. + +### 6.2 Runner registration In `bench/batteries/runner.py`: @@ -196,79 +382,147 @@ _BATTERIES = { } ``` -Existing `--all` mode picks up the new fixtures automatically once -they're in `_DEFAULT_FIXTURES`. - -### 4.3 Makefile additions +### 6.3 Make targets ```makefile -bench-5f: bootstrap ## 5F battery (Function/Finetuning/Falsification/Formulate/Feedback Loop) - $(PY) -m bench.batteries.runner --battery 5f --sub function --fixtures bench/fixtures/5f/function-v1.jsonl - $(PY) -m bench.batteries.runner --battery 5f --sub finetuning --fixtures bench/fixtures/5f/finetuning-v1.jsonl - $(PY) -m bench.batteries.runner --battery 5f --sub falsification --fixtures bench/fixtures/5f/falsification-v1.jsonl - $(PY) -m bench.batteries.runner --battery 5f --sub formulate --fixtures bench/fixtures/5f/formulate-v1.jsonl - $(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop --fixtures bench/fixtures/5f/feedback-loop-v1.jsonl +bench-5f-function: + $(PY) -m bench.batteries.runner --battery 5f --sub function \ + --fixtures bench/fixtures/5f/function-v1.jsonl -bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite +bench-5f-finetuning: + $(PY) -m bench.batteries.runner --battery 5f --sub finetuning \ + --fixtures bench/fixtures/5f/finetuning-v1.jsonl + +bench-5f-falsification: + $(PY) -m bench.batteries.runner --battery 5f --sub falsification \ + --fixtures bench/fixtures/5f/falsification-v1.jsonl + +bench-5f-formulate: + $(PY) -m bench.batteries.runner --battery 5f --sub formulate \ + --fixtures bench/fixtures/5f/formulate-v1.jsonl + +bench-5f-feedback-loop: + $(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop \ + --fixtures bench/fixtures/5f/feedback-loop-v1.jsonl + +bench-5f: bench-5f-function bench-5f-finetuning \ + bench-5f-falsification bench-5f-formulate \ + bench-5f-feedback-loop + +bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite ``` -### 4.4 Tests - -- `tests/test_bench_batteries.py` extension: positive coverage - for each new sub-battery. -- Determinism tests: each runner produces identical - `BatteryResult.fixture_digest` and `runtime_digest` across - back-to-back runs. - -### 4.5 Bumps - -- New module `bench/batteries/b_5f.py`. -- 5 new fixture files. -- New Makefile targets. -- No schema changes; no policy hash changes. +`make bench-5s5t5f` becomes the canonical Dav1DPrometheus +non-embodied AGI fitness surface. --- -## 5. Out of scope +## 7. Tests -- Phase 1b fixture expansion (target 30-50 per sub-battery). Phase - 1a's seed set lands first. -- Cross-battery composition (e.g., "score this agent across - 5S+5T+5F"). Aggregation is its own concern. -- Calibration of metric thresholds. Phase 1a measures; thresholds - for v8 selection acceptance come later. -- Auto-grading via LLM. All 5F evaluators are deterministic - predicates; LLM-as-judge is future work. +``` +test_5f_function_runs +test_5f_finetuning_runs +test_5f_falsification_runs +test_5f_formulate_runs +test_5f_feedback_loop_runs +test_5f_fixture_digest_stable +test_5f_runtime_digest_stable +test_5f_fixture_accepts_carrier_metadata +test_5f_unsupported_future_carrier_fails_cleanly +``` + +Sub-battery-specific: + +``` +test_5f_function_shape_match_evaluator +test_5f_function_pointer_set_match_evaluator +test_5f_function_threshold_on_metric_evaluator +test_5f_finetuning_requires_parent_and_child_selfmodel +test_5f_falsification_verifier_method_root_mismatch_warns +test_5f_formulate_pointer_id_set_exact_match +test_5f_feedback_loop_observation_affects_downstream_snapshot +``` --- -## 6. Risks & open questions +## 8. Out of scope -- **Function evaluator design.** "Did the system execute the - designated task" requires a per-task predicate. v1 ships a - small set of canonical evaluator types (shape-match, - pointer-set-match, threshold-on-metric). New evaluator types - land in follow-ups. -- **Finetuning needs SelfModel chains.** Realistic fixtures - require real shard data with parent→child SelfModel transitions. - Phase 1a uses synthetic chains generated in-test; Phase 1b - pulls from production shards. -- **Falsification fixtures are fragile.** A planted-error fixture - bound to a specific verifier signature may stop catching the - error if the verifier changes shape. Mitigation: tag each - Falsification fixture with the `verifier_method_root` it was - authored against; bench reports flag mismatches. +- Phase 1b fixture expansion (target 30-50 per sub-battery). +- Cross-battery composite scoring (5S + 5T + 5F aggregation is its + own concern). +- Calibration of metric thresholds for v8 selection acceptance + (depends on #000012 v8 paper). +- LLM-as-judge for any 5F task (forbidden by hard constraint). +- Multimodal Phase 1a (text/claim-lattice/memory only; multimodal + comes after #000013 + #000015 follow-ups land). +- Hidden-channel **generation**. Defensive detection only, in + Phase 1b+. --- -## 7. Status +## 9. Risks & mitigations + +| Risk | Mitigation | +|---|---| +| Function evaluator vague | v1 allows only named deterministic evaluator types: `shape_match`, `pointer_set_match`, `threshold_on_metric`. Future evaluators land in follow-ups. | +| Finetuning fakes improvement | Require parent + child SelfModel roots; require measured_value pre/post; require improvement threshold; record capital cost. | +| Falsification fixtures stale | Tag every fixture with `verifier_method_root`. Bench reports mismatch instead of false failure. | +| Formulate exact-string brittle | Structural match only: claim count + order + pointer-ID-set + claim-text approximate/sorted. | +| Feedback Loop needs substrate work | Use existing `memory_records`, `memory_branch_summaries`, `audit_events` surfaces. Phase 1a explicitly forbids new substrate. | +| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers fail/skip explicitly. | + +--- + +## 10. Closure criterion + +### Phase 1a closes when: + +1. `bench/batteries/b_5f.py` lands. +2. All five sub-battery runners are non-stub. +3. Each sub-battery has ≥10 deterministic seed fixtures. +4. Fixture metadata includes `carrier` / `domain` / `pi_star_ref` + / `loss_report_refs` / `modality_notes` fields. +5. `make bench-5f` passes end-to-end. +6. `make bench-5s5t5f` exists and runs the complete Dav1DPrometheus + suite. +7. Runtime + fixture digests are deterministic across repeated + runs. +8. No schema, governance hash, or canonicalization-version changes. +9. No LLM-as-judge anywhere in 5F runners. + +### Phase 1b keeps the ticket in-progress until: + +10. 30-50 fixtures per sub-battery. +11. Real shard-based Finetuning chains (replacing synthetic + parent→child SelfModel pairs from Phase 1a). +12. Expanded Falsification planted-error suite covering every + documented failure-motif tag. +13. Feedback Loop latency / efficiency metrics calibrated against + real workload data. +14. Threshold calibration for v8 selection acceptance handed off + to ticket #000012. + +--- + +## 11. Status **Open · awaiting go/no-go.** Larger surface than #000023 / #000024 -because 5F has five sub-batteries vs three remaining in 5S and -four in 5T. Recommended sequencing: ship Phase 1a in one push -(seed-fixture per sub-battery, ~10 each), then Phase 1b expands. +(five sub-batteries vs three / four remaining). Recommended +sequencing: ship Phase 1a in one push (seed fixtures + non-stub +runners), Phase 1b expands. -Closure criterion (Phase 1a): `bench/batteries/b_5f.py` lands -with non-stub runners for all five sub-batteries; seed fixtures -+ Makefile targets pass `make bench-5f` end-to-end. Phase 1b -keeps the ticket open in-progress until fixtures are expanded. +The synthesis statement that justifies 5F's existence as a distinct +axis: + +> **SQD gives the invariant projection and verifier model. +> Merkle-AGI v7 gives multimodal committed DAG structure. 5S gives +> sign/meaning/derivation fitness. 5T gives transfer, logic, +> triangulation, and time. 5F gives operation, adaptation, +> falsification, formulation, and feedback. Together they +> instantiate an explicit discrete state-space/time in which +> counting, mathematics, and logic become auditable operations over +> committed state — not just text generated by a latent model.** + +5F is the operational layer that turns the substrate into an +adaptive learning organism. Without it, the state-space exists +but does not improve.