docs/tickets: rewrite #000023/#000024/#000025 with cross-modality discipline
Per the three review responses (~/Downloads/RESPONSE_*) folded in 2026-05-08, the 5S / 5T / 5F tickets are corrected from "text-only with future hooks" to "carrier-aware design from day one." Phase 1 implementation stays text / claim-lattice / memory-root only, but the fixture schema MUST accommodate future visual / world / code / audio / sensor / hidden-channel-detection carriers without re-authoring. Common corrections across all three tickets: - Mandatory fixture metadata: carrier, domain, pi_star_ref, loss_report_refs, modality_notes. - Unsupported carriers MUST fail or skip explicitly with reason="unsupported_carrier" — never silently accepted. - No LLM-as-judge in any runner. - Hidden-channel work is defensive only (detection / flagging), never generation or concealment. Per-ticket headlines: #000023 — 5S Syntax / Semantics / Semiotics defined as carrier-general operations over sign-bearing representations. Semiotics gets the biggest correction: visual symbols, layout, metadata, encoded sign systems are valid carriers (Phase 1 still text-only). Synonym source policy: concept_relations.relation_kind='synonym' only for v1 positives. #000024 — 5T Vocabulary alignment with Dav1DPrometheus authoritative wording (Transfer→Transfer Learning, Truth→Truthtables, Timing→Time). Transitivity gets a typed-relation whitelist (implies, subset_of, ancestor_of, before, less_than) — not all edges transitive. Truthtables capped at N=2..4 to avoid combinatorial blowup. Time is the first sub-battery where v8 substrate (memory_root #000017 + selfmodel #000014) becomes a measurable bench target. #000025 — 5F New axis. Function/Finetuning/Falsification/Formulate/Feedback Loop. Folds in the state-space synthesis: SQD + v7 + 5S/5T/5F + arborist together instantiate a discrete state-space/time Ω_t = (W, I, C, L, MRoot, SMRoot, PRoot, BRoot, ARoot) with Ω_{t+1} = T(Ω_t, Δ_t). Counting / mathematics / logic / time emerge as auditable operations over committed state, not text from a latent model. adaptation_efficiency and feedback_efficiency metrics hook into the capital ledger (#000020) so v8 fork choice has cost-aware fitness signals. Falsification fixtures tagged with verifier_method_root so verifier shape changes warn rather than false-fail. All three tickets remain "open · awaiting go/no-go" — design-only. Implementation tickets land in follow-up commits when fox approves the corrected scope. Source: Legally Unprecedented Dav1DPrometheus (BasementAGI host, Where The mAGIc Happens). Honoring his framework.
This commit is contained in:
parent
02c7e41ef8
commit
fbd99a8d76
3 changed files with 1185 additions and 572 deletions
|
|
@ -1,230 +1,392 @@
|
|||
# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics
|
||||
# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics (carrier-aware)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Complete the 5S battery's three remaining sub-batteries —
|
||||
Syllogism, Synthesis, Semiotics — that landed as zero-task stubs in
|
||||
ticket #000021 Phase 1a. Each sub-battery gets a real fixture set
|
||||
(seed → expand) and a runner that exercises arborist's actual
|
||||
surface (claim-lattice parser, retrieval pipeline, label-swap
|
||||
behavior).
|
||||
**Audience:** fox + future bench authors + downstream v8 / v7-W work
|
||||
that needs concrete fitness targets across all 5S dimensions.
|
||||
**Hard constraint:** fixtures land deterministically (committed
|
||||
JSONL with version pin). Bench-maxing discipline (5pp signal floor,
|
||||
n≥3) applies. Each sub-battery is independently runnable via
|
||||
`make bench-5s-<sub>`. Adding sub-batteries does NOT alter Phase 1a
|
||||
behavior — Syntax + Semantics keep their existing fixture digests.
|
||||
**Revised:** 2026-05-08 (cross-modality correction folded in per
|
||||
review response)
|
||||
**Scope:** Replace the three zero-task stubs in 5S
|
||||
(Syllogism / Synthesis / Semiotics) with deterministic Phase-1b
|
||||
runners + fixture sets. Implementation stays text / claim-lattice /
|
||||
prose only; **the fixture schema is carrier-aware** so future
|
||||
phases can attach visual, world-model, code, audio, sensor, and
|
||||
hidden-channel-detection carriers without re-authoring the
|
||||
fixture format.
|
||||
**Audience:** fox + future bench authors + downstream v7-W /
|
||||
multimodal work that needs 5S evaluation across non-text carriers.
|
||||
**Hard constraint:** Phase 1a fixture digests stay pinned (Syntax
|
||||
+ Semantics keep their `v1` digests). No schema bump, no
|
||||
governance hash bump, no canonicalization-version bump. Unsupported
|
||||
non-text carriers MUST fail or skip explicitly — never silently
|
||||
accepted. No LLM-as-judge in any 5S runner.
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) named the 5Ss in his
|
||||
authoritative framework; arborist honors the original wording.
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ss: Syntax,
|
||||
Semantics, Syllogism, Synthesis, Semiotics."*
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
## 1. Corrected purpose
|
||||
|
||||
Ticket #000021 Phase 1a shipped runners + seed fixtures for 5S
|
||||
Syntax (`parse-pass`) and 5S Semantics (`equivalence`). The other
|
||||
three sub-batteries — Syllogism, Synthesis, Semiotics — landed as
|
||||
stubs that return zero-task `BatteryResult` objects so the full
|
||||
suite at least runs. Stubs are visible in `bench/batteries/b_5s.py`.
|
||||
Two coupled goals:
|
||||
|
||||
Without real implementations:
|
||||
1. **Complete the 5S battery** — three stubs become real runners
|
||||
with ≥30 deterministic fixtures each.
|
||||
2. **Make 5S carrier-general** — Syntax, Semantics, Semiotics,
|
||||
Syllogism, Synthesis are not prose-only. They are tests over
|
||||
*sign-bearing representations*. Text is only the first carrier.
|
||||
|
||||
| Gap | Failure mode |
|
||||
The architectural correction (folded in 2026-05-08):
|
||||
|
||||
> **Syntax hidden in visual or alternate media is still syntax.
|
||||
> Meaning hidden in non-text media is still semantics. A sign
|
||||
> encoded in a non-text carrier is still semiotics.**
|
||||
|
||||
#000023 lands text fixtures + text runners now. The fixture
|
||||
**schema** must accommodate future cross-carrier work without
|
||||
re-authoring.
|
||||
|
||||
---
|
||||
|
||||
## 2. Carrier-aware 5S definitions
|
||||
|
||||
| Sub-battery | Definition (carrier-general) |
|
||||
|---|---|
|
||||
| **Syllogism** | No way to bench arborist's deductive-reasoning surface. v8 selection + SelfModel capability claims have no measurable target for "stepwise entailment validity." |
|
||||
| **Synthesis** | No bench for "given a goal + a fact set, can the system produce a passing derivation?" — directly the claim-lattice + retrieval pipeline's output, which today only gets ad-hoc QA bench coverage. |
|
||||
| **Semiotics** | Label-swap invariance under controlled symbol changes is unmeasured. Drift in proper-noun handling, code-name conventions, or alphabet shifts goes undetected. |
|
||||
| **Syntax** | Formal arrangement rules of a carrier. Text: tokens, parse, markup. Code: AST, imports, indentation. Tables: rows/columns/headers. Images: layout, regions, visual grouping. World: object relations, spatial containment, event ordering. Audio: phoneme/order structure, timing. |
|
||||
| **Semantics** | Invariant meaning preserved across representation changes. Text: synonym pairs, entity equivalence. Image: a stop sign and the text "STOP" carry equivalent traffic-control meaning. Chart: an upward trend and "X increases over time" carry equivalent data semantics. World: "A inside B" represented visually, symbolically, or as coordinates. |
|
||||
| **Syllogism** | Stepwise valid deduction, given premises in any carrier. Text+table: "all rows with status=active have flag=true; row X has status=active; therefore X has flag=true." Image+rule: "A is inside B; rule says objects in B have property P; therefore A has P." World: "A left of B; B left of C; therefore A left of C." |
|
||||
| **Synthesis** | Assembling multiple warranted elements into a coherent derivation. Text+chart: derive a claim only when textual and chart evidence agree. Image+OCR+metadata: synthesize while flagging carrier conflict. World-state: synthesize a temporal claim from multiple state observations. |
|
||||
| **Semiotics** | Interpretation of signs, symbols, labels, references, frames, conventions, aliases, sign systems. Text: AMD↔"Advanced Micro Devices", Apple-fruit vs Apple-company, Orwell-frame vs literal-geography. Visual: red octagon = stop, skull icon = hazard, watermark = provenance, QR-pattern = encoded sign system, diagram-arrow = causality/flow/dependency. Future: visual steganographic carriers where surface differs from hidden encoding. |
|
||||
|
||||
### 1.1 Sub-battery semantics (per ticket #000021 §4.2)
|
||||
|
||||
- **Syllogism:** each task is a proof chain + a step index +
|
||||
expected step-valid pass/fail. Metric: step-validity-rate.
|
||||
- **Synthesis:** each task is a goal description + a fact set +
|
||||
expected derivation pass/fail. Metric: derivation-pass-rate.
|
||||
- **Semiotics:** each task is a controlled label-swap + invariant
|
||||
expected. Metric: invariance-under-swap.
|
||||
Phase 1b implements only **text / claim-lattice / prose** carriers.
|
||||
Future-carrier hooks land via fixture metadata (§3) that the v1
|
||||
runner ignores or rejects-cleanly.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
## 3. Fixture schema (carrier-aware)
|
||||
|
||||
### 2.1 Fixture sourcing for Syllogism
|
||||
Common JSONL header per task:
|
||||
|
||||
**A. Hand-curated proof chains drawn from existing claim-lattice tests.**
|
||||
arborist's `tests/test_claim_lattice.py` and `tests/test_directives.py`
|
||||
already contain stepwise-claim examples. Promoting them to
|
||||
fixtures gets us 30-50 entries cheaply.
|
||||
```json
|
||||
{
|
||||
"id": "5s-<sub>-NNN",
|
||||
"battery": "5s",
|
||||
"sub_battery": "syllogism",
|
||||
"version": "v1",
|
||||
|
||||
**B. Synthetic generators (template-based syllogisms).** Cheap;
|
||||
templates need to be committed for reproducibility.
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"loss_report_refs": [],
|
||||
"modality_notes": "Phase 1b: text-only; future carriers may include image/code/audio/world/hidden-channel.",
|
||||
|
||||
**Recommendation:** A first; add B in Phase 2 once the harness
|
||||
stabilizes.
|
||||
|
||||
### 2.2 Fixture sourcing for Synthesis
|
||||
|
||||
**A. Ablate from existing QA fixtures.** Take questions from
|
||||
`bench/qa_sweep.py` corpora; record which sources retrieve as the
|
||||
fact set; mark expected derivation status.
|
||||
|
||||
**B. Compose from claim-lattice docs.** Use docs already in shards;
|
||||
build (goal, fact_set, derivation_expected) tuples by stripping
|
||||
claims and asking whether they should re-derive.
|
||||
|
||||
**Recommendation:** A — anchors against existing QA bench so
|
||||
Synthesis-rate trends correlate with overall QA rate.
|
||||
|
||||
### 2.3 Fixture sourcing for Semiotics
|
||||
|
||||
**A. Hand-curated label-swap pairs.** Examples: `"AMD"↔"Advanced Micro
|
||||
Devices"`, `"NYC"↔"New York City"`, code-name swaps inside
|
||||
identical text.
|
||||
|
||||
**B. Generated swaps via the concept-relations table.** arborist's
|
||||
`concept_relations` already has synonym pairs from corpus
|
||||
extraction; promote them as semiotic swap fixtures.
|
||||
|
||||
**Recommendation:** B — leverages existing per-shard data, makes
|
||||
fixtures regenerable from the corpus rather than hand-pinned.
|
||||
|
||||
### 2.4 Runner integration
|
||||
|
||||
Each sub-battery is a function in `bench/batteries/b_5s.py`
|
||||
matching the existing `run_syntax`/`run_semantics` signature.
|
||||
Replace the stub `_stub_result` returns with real implementations.
|
||||
Existing test in `tests/test_bench_batteries.py
|
||||
::test_stub_sub_batteries_return_zero` will need to flip to
|
||||
non-zero assertions once these land.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
Land in this order:
|
||||
|
||||
1. **Syllogism** first — purely lexical (claim-lattice traversal),
|
||||
no shard reads, fastest implementation.
|
||||
2. **Semiotics** second — depends on `concept_relations` lookup
|
||||
but no LLM call; deterministic.
|
||||
3. **Synthesis** third — requires shard reads + retrieval pipeline;
|
||||
slowest, biggest surface, highest variance. Defer until
|
||||
Syllogism + Semiotics stabilize the harness.
|
||||
|
||||
Phase 1b closure = all three sub-batteries shipping with at minimum
|
||||
**30 deterministic fixtures each** and runners that integrate
|
||||
cleanly into `make bench-5s`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Fixture files
|
||||
|
||||
```
|
||||
bench/fixtures/5s/syllogism-v1.jsonl # 30+ tasks
|
||||
bench/fixtures/5s/synthesis-v1.jsonl # 30+ tasks
|
||||
bench/fixtures/5s/semiotics-v1.jsonl # 30+ tasks
|
||||
"task": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
Each fixture file carries the standard `_meta` head:
|
||||
Field semantics:
|
||||
|
||||
| Field | Phase 1b allowed values | Future values |
|
||||
|---|---|---|
|
||||
| `carrier` | `text`, `claim_lattice`, `prose` | `code`, `table`, `html`, `image`, `audio`, `layout`, `world`, `mixed`, `hidden_channel` |
|
||||
| `domain` | `prose`, `wikitext`, `claim_lattice` | `python_ast`, `chart`, `spatial_state`, `propositional_logic`, etc. |
|
||||
| `pi_star_ref` | `wikitext-base@v1`, `claim-lattice@v1` | future π* registry keys |
|
||||
| `loss_report_refs` | `[]` | optional projection-loss record refs (#000022) |
|
||||
|
||||
Unsupported `carrier` / `domain` values → runner fails task
|
||||
explicitly with `reason="unsupported_carrier"` or skips with same
|
||||
reason logged. Never silently accepted.
|
||||
|
||||
---
|
||||
|
||||
## 4. Per-sub-battery implementation
|
||||
|
||||
### 4.1 Syllogism
|
||||
|
||||
**Metric:** `step_validity_rate = valid_steps / total_steps`.
|
||||
|
||||
**Positive fixture:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5s-syllogism-001",
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"premises": [
|
||||
{"claim": "All mammals are animals.", "evidence_ids": ["E1"]},
|
||||
{"claim": "All dogs are mammals.", "evidence_ids": ["E2"]}
|
||||
],
|
||||
"candidate_step": {
|
||||
"claim": "All dogs are animals.",
|
||||
"uses": ["E1", "E2"]
|
||||
},
|
||||
"rule": "categorical_transitivity",
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Negative fixture:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5s-syllogism-NEG-001",
|
||||
"premises": [
|
||||
{"claim": "All dogs are animals.", "evidence_ids": ["E1"]},
|
||||
{"claim": "All cats are animals.", "evidence_ids": ["E2"]}
|
||||
],
|
||||
"candidate_step": {
|
||||
"claim": "All animals are dogs.",
|
||||
"uses": ["E1", "E2"]
|
||||
},
|
||||
"rule": "invalid_converse",
|
||||
"expected": "fail"
|
||||
}
|
||||
```
|
||||
|
||||
**Runner behavior:** load fixture → parse premises into
|
||||
claim-lattice → parse candidate step → check referenced prior
|
||||
claims exist → check step is licensed by declared rule → return
|
||||
pass/fail. Deterministic, no LLM judge.
|
||||
|
||||
### 4.2 Synthesis
|
||||
|
||||
**Metric:** `derivation_pass_rate = passed / total`.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5s-synthesis-001",
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"goal": "Derive whether the source supports the release-date claim.",
|
||||
"fact_set": [
|
||||
{"id": "F1", "text": "Back to the Future was released on July 3, 1985."},
|
||||
{"id": "F2", "text": "The film was directed by Robert Zemeckis."}
|
||||
],
|
||||
"expected_derivation": [
|
||||
{"claim": "Back to the Future was released on July 3, 1985.", "uses": ["F1"]}
|
||||
],
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Future-carrier hook:** fact-set entries may later include
|
||||
non-text carriers (`carrier: "chart"`, `carrier: "image"`). v1
|
||||
runner rejects them as unsupported.
|
||||
|
||||
**Variance control:** if a Synthesis task uses arborist's retrieval
|
||||
pipeline, the fixture meta MUST pin `shard_set`, `retrieval_mode`,
|
||||
and `expected_fact_ids`. Bench-maxing 5pp signal floor handles
|
||||
residual noise; pinned fact IDs eliminate the largest variance
|
||||
source.
|
||||
|
||||
### 4.3 Semiotics
|
||||
|
||||
**Metric:** `invariance_under_swap = invariant_passes / total`.
|
||||
|
||||
**Synonym-swap fixture (positive):**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5s-semiotics-001",
|
||||
"carrier": "text",
|
||||
"domain": "prose",
|
||||
"pi_star_ref": "wikitext-base@v1",
|
||||
"source_text": "AMD released a new GPU.",
|
||||
"swap": {
|
||||
"from": "AMD",
|
||||
"to": "Advanced Micro Devices",
|
||||
"relation_kind": "synonym"
|
||||
},
|
||||
"invariant_fields": ["entity", "predicate", "object"],
|
||||
"expected_invariant": true
|
||||
}
|
||||
```
|
||||
|
||||
**Reference-frame fixture:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5s-semiotics-002",
|
||||
"source_text": "Oceania has always been at war with Eastasia.",
|
||||
"frame": "orwell_1984",
|
||||
"literal_frame_expected": false,
|
||||
"reference_frame_expected": true,
|
||||
"expected_invariant": true
|
||||
}
|
||||
```
|
||||
|
||||
**Future-carrier hook (documented in fixture, not executed in v1):**
|
||||
|
||||
```json
|
||||
{
|
||||
"future_carrier_equivalent": {
|
||||
"carrier": "image",
|
||||
"description": "symbolic sign or visual glyph equivalent may later replace text label"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Hidden-channel safety framing.** Phase 1b does NOT generate,
|
||||
hide, or recover covert messages. Future hidden-channel fixtures
|
||||
MUST be framed as **defensive detection / flagging** only:
|
||||
|
||||
```json
|
||||
{
|
||||
"carrier": "image",
|
||||
"sub_battery": "semiotics",
|
||||
"hidden_channel_expected": true,
|
||||
"task_kind": "detect_unexpected_sign_channel",
|
||||
"expected": "flag"
|
||||
}
|
||||
```
|
||||
|
||||
This keeps the benchmark defensive. The capability tested is
|
||||
"can the system detect a suspicious carrier" — not "can the
|
||||
system encode a covert message."
|
||||
|
||||
**Source policy for v1 fixtures:** Semiotics fixtures use
|
||||
`concept_relations.relation_kind = 'synonym'` only. Negative
|
||||
fixtures pair with non-equivalent swaps explicitly tagged.
|
||||
|
||||
---
|
||||
|
||||
## 5. Required file changes
|
||||
|
||||
### 5.1 New fixture files
|
||||
|
||||
```
|
||||
bench/fixtures/5s/syllogism-v1.jsonl # ≥30 deterministic tasks
|
||||
bench/fixtures/5s/synthesis-v1.jsonl # ≥30 deterministic tasks
|
||||
bench/fixtures/5s/semiotics-v1.jsonl # ≥30 deterministic tasks
|
||||
```
|
||||
|
||||
Each carries the standard `_meta` head:
|
||||
|
||||
```json
|
||||
{"_meta":{"battery":"5s","sub_battery":"syllogism","version":"v1","task_count":30}}
|
||||
```
|
||||
|
||||
### 4.2 Runner additions
|
||||
Phase 1a fixture digests (`syntax-v1.jsonl`, `semantics-v1.jsonl`)
|
||||
stay pinned.
|
||||
|
||||
In `bench/batteries/b_5s.py`, replace stub bodies:
|
||||
### 5.2 Runner functions
|
||||
|
||||
```python
|
||||
def run_syllogism(fixtures_path: Path) -> BatteryResult:
|
||||
# for each task: parse claim chain, walk step_index, assert
|
||||
# the claim references valid evidence + the prior step entails
|
||||
# this one under claim-lattice semantics. Metric:
|
||||
# step_validity_rate = pass_count / (pass + fail).
|
||||
...
|
||||
```
|
||||
In `bench/batteries/b_5s.py`, replace stub bodies of
|
||||
`run_syllogism`, `run_synthesis`, `run_semiotics`. Existing
|
||||
`run_syntax`, `run_semantics`, `_runtime_digest` stay untouched.
|
||||
|
||||
Same shape for `run_synthesis`, `run_semiotics`. Existing
|
||||
`_runtime_digest` helper covers the runtime fingerprint.
|
||||
|
||||
### 4.3 Makefile additions
|
||||
### 5.3 Make targets
|
||||
|
||||
```makefile
|
||||
bench-5s-syllogism: bootstrap
|
||||
bench-5s-syllogism: ## 5S Syllogism (step-validity-rate)
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub syllogism \
|
||||
--fixtures bench/fixtures/5s/syllogism-v1.jsonl
|
||||
|
||||
bench-5s-synthesis: bootstrap
|
||||
bench-5s-synthesis: ## 5S Synthesis (derivation-pass-rate)
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub synthesis \
|
||||
--fixtures bench/fixtures/5s/synthesis-v1.jsonl
|
||||
|
||||
bench-5s-semiotics: bootstrap
|
||||
bench-5s-semiotics: ## 5S Semiotics (invariance-under-swap)
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub semiotics \
|
||||
--fixtures bench/fixtures/5s/semiotics-v1.jsonl
|
||||
```
|
||||
|
||||
Existing `bench-5s` aggregate target picks them up via `--all`
|
||||
once the default fixture map in `runner.py` includes them.
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- Replace `test_stub_sub_batteries_return_zero` parametrizations
|
||||
for the three sub-batteries with positive coverage:
|
||||
`test_5s_syllogism_runs`, `test_5s_synthesis_runs`,
|
||||
`test_5s_semiotics_runs`.
|
||||
- Add fixture-digest stability test for each new fixture file.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New fixture files (additive). Phase 1a fixture digests stay
|
||||
pinned.
|
||||
- Runner-stub replacements in `b_5s.py`.
|
||||
- New Makefile targets.
|
||||
- No schema changes. No `governance_policy_hash` changes.
|
||||
The existing `bench-5s` aggregate target picks them up via
|
||||
`--all` once `_DEFAULT_FIXTURES` in `runner.py` includes them.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
## 6. Tests
|
||||
|
||||
- 5T Phase 1b (Triangulation, Truthtables, Transitivity, Time —
|
||||
ticket #000024).
|
||||
- 5F battery (Function, Finetuning, Falsification, Formulate,
|
||||
Feedback Loop — ticket #000025).
|
||||
- 5R battery (workspace operators) — depends on SelfModel
|
||||
(#000014, landed) + memory_root (#000017, landed); reserved as
|
||||
Phase 2 of #000021.
|
||||
Replace stub-zero parametrizations for the three sub-batteries.
|
||||
|
||||
Required positive coverage:
|
||||
|
||||
```
|
||||
test_5s_syllogism_runs
|
||||
test_5s_synthesis_runs
|
||||
test_5s_semiotics_runs
|
||||
test_5s_syllogism_fixture_digest_stable
|
||||
test_5s_synthesis_fixture_digest_stable
|
||||
test_5s_semiotics_fixture_digest_stable
|
||||
test_5s_phase1a_digest_unchanged # guard against regression
|
||||
```
|
||||
|
||||
Modality-safety tests (carrier-schema future-readiness):
|
||||
|
||||
```
|
||||
test_5s_fixture_accepts_carrier_metadata
|
||||
test_5s_text_runner_rejects_unsupported_nontext_carrier_cleanly
|
||||
test_5s_semiotics_synonym_swap_preserves_invariant_fields
|
||||
test_5s_semiotics_bad_swap_fails
|
||||
```
|
||||
|
||||
The last group enforces that the schema accepts carrier metadata
|
||||
today even though only text is executed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Out of scope
|
||||
|
||||
- Cross-domain π* composition for 5S (covered by ticket #000015,
|
||||
already landed).
|
||||
- Multimodal 5S execution (depends on #000013 spatial-temporal
|
||||
substrate + π* image / audio / world canonicalizers; future
|
||||
tickets).
|
||||
- 5T Phase 1b (ticket #000024).
|
||||
- 5F battery (ticket #000025).
|
||||
- 5R battery — depends on SelfModel (#000014, landed) +
|
||||
memory_root (#000017, landed); reserved as Phase 2 of #000021.
|
||||
- Cross-corpus expansion (Phase 3 of #000021).
|
||||
- Hidden-channel **generation** or **concealment** tooling.
|
||||
Detection-only is the boundary.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
## 8. Risks & mitigations
|
||||
|
||||
- **Syllogism fixture quality.** Hand-curated chains can over-fit
|
||||
to arborist's current claim-lattice parser. Mitigation: pull
|
||||
from at least three different test files so coverage spans
|
||||
multiple authoring styles.
|
||||
- **Synthesis variance.** Depends on retrieval which has
|
||||
per-shard variance. Mitigation: pin shard set in the fixture
|
||||
meta; bench-maxing 5pp signal floor accounts for noise.
|
||||
- **Semiotics blast radius.** A bad swap fixture could mask
|
||||
genuine drift if the swap is too aggressive (e.g., swaps that
|
||||
also change semantic meaning, not just surface). Mitigation:
|
||||
curate from `concept_relations` of kind `synonym` only — those
|
||||
are corpus-derived equivalences with known semantic preservation.
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Syllogism overfits to toy chains | Pull from ≥3 different test-source families; include invalid converses, missing-premise, unsupported-evidence-pointer failures. |
|
||||
| Semiotics swaps accidentally change meaning | Use only `concept_relations.relation_kind='synonym'` for v1 positives; include explicit non-equivalent negative swaps. |
|
||||
| Synthesis retrieval variance | Pin `shard_set` + `retrieval_mode` + `expected_fact_ids` in fixture meta; compare derivation shape, not exact prose. |
|
||||
| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers rejected cleanly in v1; multimodal execution deferred to #000013 / #000015 follow-ups. |
|
||||
| Steganography → dual-use tooling | v1 does not generate, hide, or recover covert messages. Future fixtures framed as defensive detection / flagging. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
## 9. Closure criterion
|
||||
|
||||
#000023 closes when:
|
||||
|
||||
1. `run_syllogism`, `run_synthesis`, `run_semiotics` are all non-stub.
|
||||
2. Each ships ≥30 deterministic fixtures with stable `fixture_digest`.
|
||||
3. Fixtures include `carrier` / `domain` / `pi_star_ref` /
|
||||
`loss_report_refs` / `modality_notes` fields.
|
||||
4. Unsupported non-text carriers fail/skip explicitly with
|
||||
`reason="unsupported_carrier"`.
|
||||
5. `make bench-5s` runs all five 5S sub-batteries (Syntax,
|
||||
Semantics, Syllogism, Synthesis, Semiotics).
|
||||
6. Phase 1a fixture digests (`syntax-v1.jsonl`,
|
||||
`semantics-v1.jsonl`) unchanged.
|
||||
7. No schema, governance hash, or canonicalization-version changes.
|
||||
8. Tests cover runner execution, fixture-digest stability, and
|
||||
carrier-schema future-readiness.
|
||||
|
||||
---
|
||||
|
||||
## 10. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Three sub-batteries × ~30 fixtures
|
||||
× one runner each ≈ 600-800 lines of code + fixtures. Estimated
|
||||
mid-sized session.
|
||||
× one runner each ≈ 800-1000 lines code + fixtures. Mid-sized
|
||||
session.
|
||||
|
||||
Closure criterion: all three runners non-stub, fixture files
|
||||
landed with version pins, `make bench-5s` runs all five 5S
|
||||
sub-batteries (Syntax, Semantics, Syllogism, Synthesis, Semiotics)
|
||||
and emits a JSON result per sub-battery. Existing Phase 1a
|
||||
fixtures keep their digests.
|
||||
The architectural correction from the 2026-05-08 review:
|
||||
**Semiotics, Syntax, and Semantics are defined over sign-bearing
|
||||
carriers, not just written language.** Phase 1b ships the text
|
||||
implementation; the fixture schema preserves the future path
|
||||
into spatial-temporal world models, multimodal AGI evaluation,
|
||||
and defensive hidden-channel detection.
|
||||
|
|
|
|||
|
|
@ -1,160 +1,318 @@
|
|||
# Ticket #000024 — 5T Phase 1b: complete sub-batteries + align vocabulary with Dav1DPrometheus
|
||||
# Ticket #000024 — 5T Phase 1b: vocabulary alignment + carrier-aware completion
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Complete the 5T battery — rename existing sub-batteries
|
||||
to match Dav1DPrometheus's authoritative vocabulary
|
||||
(Transfer → Transfer Learning, Truth → Truthtables, Timing →
|
||||
Time), and ship real implementations + fixtures for the four
|
||||
remaining stub sub-batteries (Triangulation, Truthtables,
|
||||
Transitivity, Time).
|
||||
**Audience:** fox + future bench authors + anyone composing a v8
|
||||
fitness target across the 5T axes.
|
||||
**Hard constraint:** the rename is a NEW fixture-version landing
|
||||
(`v1` → `v2`); existing `transfer-v1.jsonl` stays in place under
|
||||
its sub-battery name `transfer` so #000021 Phase 1a digests don't
|
||||
shift retroactively. New work uses the renamed identifiers.
|
||||
**Revised:** 2026-05-08 (cross-modality correction folded in per
|
||||
review response)
|
||||
**Scope:** Two coupled jobs:
|
||||
1. Align 5T vocabulary with Dav1DPrometheus's authoritative
|
||||
formulation (Transfer → Transfer Learning, Truth → Truthtables,
|
||||
Timing → Time).
|
||||
2. Replace the four 5T zero-task stubs (Triangulation, Truthtables,
|
||||
Transitivity, Time) with carrier-aware Phase-1b runners.
|
||||
**Audience:** fox + future bench authors + downstream v8 / v7-W /
|
||||
multimodal work needing fitness targets across 5T axes.
|
||||
**Hard constraint:** Phase 1a `transfer-v1.jsonl` digest stays
|
||||
pinned; new work uses `v2` fixture filenames and renamed sub-battery
|
||||
identifiers. No schema bump, no governance hash bump, no
|
||||
canonicalization-version bump. Unsupported non-text carriers MUST
|
||||
fail or skip explicitly. No LLM-as-judge in any 5T runner.
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ts: Transitivity,
|
||||
Transfer Learning, Triangulation, Truthtables, Time."*
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
## 1. Corrected purpose
|
||||
|
||||
Two coupled gaps:
|
||||
**Vocabulary correction is not cosmetic.** Each rename pins a more
|
||||
operationally precise concept:
|
||||
|
||||
### 1.1 Vocabulary drift from authoritative framework
|
||||
|
||||
The SQD whitepaper used: **Transfer · Triangulate · Timing ·
|
||||
Transitivity · Truth.**
|
||||
|
||||
Dav1DPrometheus's published framework uses: **Transitivity ·
|
||||
**Transfer Learning** · Triangulation · **Truthtables** · **Time**.**
|
||||
|
||||
Three differences:
|
||||
|
||||
| arborist (current) | Dav1DPrometheus | Why his is right |
|
||||
| arborist Phase 1a | Dav1DPrometheus | Why his is right |
|
||||
|---|---|---|
|
||||
| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; "Transfer Learning" pins the concrete ML primitive. |
|
||||
| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the concrete logical-scenario coverage primitive. |
|
||||
| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, which is the actual semantic. |
|
||||
| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; pins the concrete ML primitive (cross-task / cross-domain method preservation). |
|
||||
| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the explicit logical-state coverage primitive. |
|
||||
| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, lineage, memory, stale-state handling. |
|
||||
|
||||
arborist landed Phase 1a with the SQD-whitepaper names. Per memory
|
||||
note `dav1dprometheus_framework.md`, his wording is authoritative —
|
||||
arborist honors it.
|
||||
The architectural correction (folded in 2026-05-08):
|
||||
|
||||
### 1.2 Four stub sub-batteries return zero tasks
|
||||
> **5T must not be permanently language-only or prose-only.
|
||||
> Transitivity, Transfer Learning, Triangulation, Truthtables, and
|
||||
> Time must eventually operate across text, code, tables, images,
|
||||
> audio, sensor streams, spatial state, world-model state, and
|
||||
> hidden or alternate media carriers.**
|
||||
|
||||
Same as 5S (ticket #000023): Triangulation, Truthtables,
|
||||
Transitivity, Time all currently return zero-task `BatteryResult`
|
||||
objects via `_stub_result` in `bench/batteries/b_5t.py`. Without
|
||||
real implementations:
|
||||
|
||||
| Sub-battery | Failure mode under stubs |
|
||||
|---|---|
|
||||
| **Triangulation** | No bench for "do independent verifiers agree?" — directly the v7 §11.4 cross-source consistency claim. |
|
||||
| **Truthtables** | No coverage of "for these N propositional variables, does the system handle all 2^N cases?" — gates v8 selection's ability to certify logical-completeness claims. |
|
||||
| **Transitivity** | No bench for "if A→B and B→C, does the system derive A→C?" — multi-step entailment is a SelfModel capability claim with no current measurement. |
|
||||
| **Time** | No bench for "does the system maintain temporal context across queries?" — directly relates to memory_root + SelfModel's lineage chain, both shipped but unbenchmarked. |
|
||||
Phase 1b implements **text / claim-lattice / memory-root** carriers
|
||||
only. The fixture schema is carrier-aware so future phases can
|
||||
attach code / image / audio / sensor / world canonicalizers via
|
||||
the π* registry without re-authoring fixtures.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
## 2. Carrier-aware 5T definitions
|
||||
|
||||
### 2.1 Rename strategy
|
||||
| Sub-battery | Definition (carrier-general) | Phase 1b carrier |
|
||||
|---|---|---|
|
||||
| **Transfer Learning** | Whether a learned structure or method transfers across a task, domain, or carrier. Future: text rule → table task; diagram relation → textual derivation; world-state pattern → claim-lattice conclusion. | text / claim_lattice |
|
||||
| **Triangulation** | Independent evidence, verifier, or modality paths converging on the same claim state. Future: text + table + chart agreement; image/OCR + caption + metadata; sensor + event log + textual report. | claim_lattice (existing four verifier strategies) |
|
||||
| **Transitivity** | Relation chains preserving valid inference across steps. Future: spatial + visual relations; code dependency + package metadata; sensor temporal-ordering. | claim_lattice (typed relation graph) |
|
||||
| **Truthtables** | Exhaustive logical-state coverage. Future: visual state toggles; world-state propositions; sensor Boolean predicates; tool precondition matrices. | propositional_logic (N=2..4 variables) |
|
||||
| **Time** | Temporal-context maintenance, lineage, memory, stale-state handling. Future: video frames; sensor streams; world-state transitions; tool/action DAG timelines. | memory_snapshot / selfmodel_snapshot |
|
||||
|
||||
**A. Hard rename in place.** Update `b_5t.py` function names and
|
||||
fixture filenames; existing Phase 1a digests change. Cleanest
|
||||
forward but invalidates Phase 1a results.
|
||||
|
||||
**B. Coexist v1 + v2.** Keep `transfer-v1.jsonl` + `run_transfer`
|
||||
function in place; add `transfer-learning-v2.jsonl` +
|
||||
`run_transfer_learning`. New work uses v2 names. Old `bench-5t`
|
||||
target still runs against v1.
|
||||
|
||||
**C. Soft rename (alias).** Keep file names; add alias mapping in
|
||||
the runner so `--sub transfer-learning` resolves to `run_transfer`.
|
||||
Code stays ugly forever.
|
||||
|
||||
**Recommendation:** B. Phase 1a was honest at the time it landed
|
||||
(SQD whitepaper was the only source then); we don't retroactively
|
||||
break it. New work explicitly uses Dav1DPrometheus's vocabulary.
|
||||
|
||||
### 2.2 Fixture sourcing
|
||||
|
||||
- **Triangulation** — replay arborist's existing four-strategy
|
||||
verifier (quote/span/entity/paraphrase) on a fixture set;
|
||||
metric is the per-task agreement rate. Already-shipped verifier
|
||||
surface; fixtures are easy.
|
||||
- **Truthtables** — small-N propositional fixtures (N=2..4); each
|
||||
fixture lists all 2^N input combinations + the expected output;
|
||||
arborist's claim-lattice has to evaluate every row. New surface
|
||||
(currently arborist doesn't do propositional eval); requires a
|
||||
thin wrapper.
|
||||
- **Transitivity** — multi-step claim chains pulled from existing
|
||||
claim-lattice tests; metric is full-chain-pass-rate.
|
||||
- **Time** — multi-query sequences against the same shard set;
|
||||
metric is "does memory_root snapshot N+1 preserve information
|
||||
from snapshot N?" Direct integration with memory-root (#000017,
|
||||
landed).
|
||||
|
||||
### 2.3 Cross-reference to memory-root and SelfModel
|
||||
|
||||
The Time sub-battery implementation reads memory_records and
|
||||
selfmodel_records to assert temporal-context preservation. This
|
||||
makes #000024 the first ticket where the v8 substrate
|
||||
(SelfModel + memory_root) gets a measurable bench target, not
|
||||
just an audit-chain artifact.
|
||||
Time is the **first sub-battery where v8 substrate components**
|
||||
(SelfModel #000014 + memory_root #000017) become measurable bench
|
||||
targets. That is structurally significant: until #000024, those
|
||||
substrates were audit-chain artifacts only; now they have a
|
||||
fitness signal.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
## 3. Fixture schema (carrier-aware)
|
||||
|
||||
Two-phase landing:
|
||||
```json
|
||||
{
|
||||
"id": "5t-<sub>-NNN",
|
||||
"battery": "5t",
|
||||
"sub_battery": "time",
|
||||
"version": "v1",
|
||||
|
||||
**Phase 1b.1 — Rename + cosmetic alignment.** Add v2 fixture file
|
||||
names + new sub-battery aliases. Keep v1 in place. ~100 lines.
|
||||
"carrier": "memory_snapshot",
|
||||
"domain": "memory_root",
|
||||
"pi_star_ref": "pi_memory_v1",
|
||||
"loss_report_refs": [],
|
||||
"modality_notes": "Phase 1b: text/claim-lattice/memory-root only; future carriers may include code/table/image/audio/sensor/world.",
|
||||
|
||||
**Phase 1b.2 — Implement four sub-batteries.** Triangulation
|
||||
first (existing surface), then Transitivity (lexical), then
|
||||
Truthtables (new wrapper), then Time (memory-root-integrated).
|
||||
~600-800 lines.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Fixture files (new)
|
||||
|
||||
```
|
||||
bench/fixtures/5t/transfer-learning-v2.jsonl # rename of transfer-v1
|
||||
bench/fixtures/5t/triangulation-v1.jsonl
|
||||
bench/fixtures/5t/truthtables-v1.jsonl
|
||||
bench/fixtures/5t/transitivity-v1.jsonl
|
||||
bench/fixtures/5t/time-v1.jsonl
|
||||
"task": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
`transfer-v1.jsonl` stays in place; `bench-5t-transfer` still runs
|
||||
it. Phase 1a tests stay green.
|
||||
Phase 1b allowed carriers: `text`, `claim_lattice`,
|
||||
`memory_snapshot`, `selfmodel_snapshot`, `verifier_strategies`,
|
||||
`propositional_logic`, `relation_graph`. Other values →
|
||||
`reason="unsupported_carrier"`.
|
||||
|
||||
### 4.2 Runner additions
|
||||
---
|
||||
|
||||
In `bench/batteries/b_5t.py`, real implementations replace stubs:
|
||||
## 4. Per-sub-battery design
|
||||
|
||||
- `run_transfer_learning(fixtures_path)` — same logic as
|
||||
`run_transfer`; alias.
|
||||
- `run_triangulation(fixtures_path)` — for each task, call
|
||||
arborist's four verifier strategies on (claim, evidence) pairs
|
||||
and report agreement rate.
|
||||
- `run_truthtables(fixtures_path)` — for each task, evaluate the
|
||||
claim-lattice over 2^N input combinations.
|
||||
- `run_transitivity(fixtures_path)` — chase A→B, B→C through the
|
||||
claim-lattice; assert C derives.
|
||||
- `run_time(fixtures_path)` — multi-query memory_root preservation;
|
||||
reads `memory_records` to assert continuity.
|
||||
### 4.1 Transfer Learning (rename of Phase 1a `transfer`)
|
||||
|
||||
### 4.3 Makefile additions
|
||||
**Metric:** `transfer_learning_success_rate`.
|
||||
|
||||
**Backward compat:** Keep `transfer-v1.jsonl`, `run_transfer`,
|
||||
`bench-5t-transfer` in place — Phase 1a digest stays pinned. Add
|
||||
`transfer-learning-v2.jsonl`, `run_transfer_learning`,
|
||||
`bench-5t-transfer-learning` for new work.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-transfer-learning-001",
|
||||
"version": "v2",
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"source_task": {
|
||||
"pattern": "A supports B when cited span contains anchor B.",
|
||||
"example": "Release-date claim with date anchor."
|
||||
},
|
||||
"target_task": {
|
||||
"pattern": "Founder-date claim with date anchor.",
|
||||
"expected_transfer": true
|
||||
},
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 Triangulation
|
||||
|
||||
**Metric:** `triangulation_agreement_rate`,
|
||||
`strategy_disagreement_rate`.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-triangulation-001",
|
||||
"carrier": "claim_lattice",
|
||||
"domain": "verifier_strategies",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"claim": "Back to the Future was released on July 3, 1985.",
|
||||
"evidence": ["E1", "E2"],
|
||||
"strategies": ["quote", "span", "entity", "paraphrase"],
|
||||
"expected_agreement_min": 0.75
|
||||
}
|
||||
```
|
||||
|
||||
**Runner:** invokes arborist's existing four verifier strategies
|
||||
on (claim, evidence) pairs, computes pairwise agreement, passes
|
||||
when agreement ≥ `expected_agreement_min`.
|
||||
|
||||
**Future-modality hook:**
|
||||
|
||||
```json
|
||||
{
|
||||
"future_modality_paths": [
|
||||
{"carrier": "table", "role": "structured evidence"},
|
||||
{"carrier": "image", "role": "visual confirmation"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
v1 runner ignores; fixture is forward-compatible.
|
||||
|
||||
### 4.3 Transitivity
|
||||
|
||||
**Metric:** `full_chain_pass_rate`.
|
||||
|
||||
**Critical correction:** not all relations are transitive. Phase
|
||||
1b ships a typed-relation-whitelist:
|
||||
|
||||
```
|
||||
implies
|
||||
subset_of
|
||||
ancestor_of
|
||||
before
|
||||
less_than
|
||||
```
|
||||
|
||||
Runners reject mixed-relation chains and non-transitive relations
|
||||
(`related_to`, `causes` without domain constraints, etc.).
|
||||
|
||||
**Positive fixture:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-transitivity-001",
|
||||
"carrier": "claim_lattice",
|
||||
"domain": "relation_graph",
|
||||
"pi_star_ref": "pi_relation_graph_v1",
|
||||
"edges": [
|
||||
{"from": "A", "to": "B", "relation": "implies"},
|
||||
{"from": "B", "to": "C", "relation": "implies"}
|
||||
],
|
||||
"query": {"from": "A", "to": "C", "relation": "implies"},
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Negative fixture:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-transitivity-NEG-001",
|
||||
"edges": [
|
||||
{"from": "A", "to": "B", "relation": "related_to"},
|
||||
{"from": "B", "to": "C", "relation": "causes"}
|
||||
],
|
||||
"query": {"from": "A", "to": "C", "relation": "causes"},
|
||||
"expected": "fail",
|
||||
"reason": "mixed relation types do not license transitive closure"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.4 Truthtables
|
||||
|
||||
**Metric:** `truth_table_coverage_rate`, `row_accuracy_rate`.
|
||||
|
||||
**Constraint:** N=2..4 variables only (avoids combinatorial
|
||||
blowup; keeps logical coverage real).
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-truthtables-001",
|
||||
"carrier": "claim_lattice",
|
||||
"domain": "propositional_logic",
|
||||
"pi_star_ref": "pi_truth_table_v1",
|
||||
"variables": ["A", "B"],
|
||||
"expression": "A AND B",
|
||||
"rows": [
|
||||
{"inputs": {"A": false, "B": false}, "expected": false},
|
||||
{"inputs": {"A": false, "B": true}, "expected": false},
|
||||
{"inputs": {"A": true, "B": false}, "expected": false},
|
||||
{"inputs": {"A": true, "B": true}, "expected": true}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Runner:** parse expression → enumerate rows → evaluate each
|
||||
deterministically → fixture fails if any row mismatches.
|
||||
Deterministic parser; no LLM.
|
||||
|
||||
### 4.5 Time
|
||||
|
||||
**Metric:** `temporal_context_preservation_rate`,
|
||||
`stale_marking_rate`, `memory_lineage_pass_rate`.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5t-time-001",
|
||||
"carrier": "memory_snapshot",
|
||||
"domain": "memory_root",
|
||||
"pi_star_ref": "pi_memory_v1",
|
||||
"snapshots": [
|
||||
{"snapshot_id": "S0", "memory_root": "M0", "facts": ["X is true as of t0"]},
|
||||
{"snapshot_id": "S1", "memory_root": "M1", "facts": ["X is revised as of t1"]}
|
||||
],
|
||||
"expected": {
|
||||
"preserves_prior": true,
|
||||
"marks_stale": ["X is true as of t0"],
|
||||
"current_root": "M1"
|
||||
},
|
||||
"policy_assumption": {
|
||||
"memory_root_binding": "advisory",
|
||||
"selfmodel_integration": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Runner:** reads `memory_records` (and `selfmodel_records` where
|
||||
needed) → compares snapshot N and N+1 → asserts temporal
|
||||
preservation, stale marking, lineage continuity.
|
||||
|
||||
**Future-time-carriers (documented, not executed in v1):**
|
||||
|
||||
```json
|
||||
{
|
||||
"future_time_carriers": [
|
||||
{"carrier": "video", "domain": "frame_sequence"},
|
||||
{"carrier": "sensor", "domain": "time_series"},
|
||||
{"carrier": "world", "domain": "state_transition_graph"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Required file changes
|
||||
|
||||
### 5.1 Fixture files
|
||||
|
||||
```
|
||||
bench/fixtures/5t/transfer-learning-v2.jsonl # new ≥30 tasks
|
||||
bench/fixtures/5t/triangulation-v1.jsonl # new ≥30 tasks
|
||||
bench/fixtures/5t/truthtables-v1.jsonl # new ≥30 tasks (N=2..4)
|
||||
bench/fixtures/5t/transitivity-v1.jsonl # new ≥30 tasks (whitelist)
|
||||
bench/fixtures/5t/time-v1.jsonl # new ≥30 tasks (memory)
|
||||
|
||||
bench/fixtures/5t/transfer-v1.jsonl # UNCHANGED (Phase 1a digest pinned)
|
||||
```
|
||||
|
||||
### 5.2 Runner functions
|
||||
|
||||
In `bench/batteries/b_5t.py`:
|
||||
|
||||
- Keep `run_transfer` (Phase 1a) intact.
|
||||
- Add `run_transfer_learning`, `run_triangulation`,
|
||||
`run_truthtables`, `run_transitivity`, `run_time` as non-stub
|
||||
implementations.
|
||||
|
||||
### 5.3 Make targets
|
||||
|
||||
```makefile
|
||||
bench-5t-transfer-learning: bootstrap
|
||||
|
|
@ -178,61 +336,100 @@ bench-5t-time: bootstrap
|
|||
--fixtures bench/fixtures/5t/time-v1.jsonl
|
||||
```
|
||||
|
||||
Existing `bench-5t` aggregate keeps `transfer` (v1) for back-compat.
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- Add `test_5t_transfer_learning_alias` — confirms new name resolves
|
||||
to same surface as `transfer`.
|
||||
- Add positive tests for each new sub-battery.
|
||||
- Update stub-test parametrization to remove the four sub-batteries
|
||||
that get real implementations.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New fixture files (additive).
|
||||
- New runner functions; old ones (`run_transfer`,
|
||||
`run_triangulate`, etc.) stay in place for back-compat.
|
||||
- New Makefile targets.
|
||||
- No schema changes.
|
||||
`bench-5t` aggregate keeps `transfer` (v1) for backward
|
||||
compatibility. New canonical aggregate is `bench-5t-v2` covering
|
||||
the five Dav1DPrometheus sub-batteries; or operators wire
|
||||
`bench-5t` to the v2 names once the rename has stabilized.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
## 6. Tests
|
||||
|
||||
- 5S Phase 1b — ticket #000023.
|
||||
- 5F battery (entirely new) — ticket #000025.
|
||||
- 5R battery (workspace operators) — Phase 2 of #000021.
|
||||
- Retiring the v1 names. Once v2 is the standard, deprecate v1
|
||||
in a follow-up ticket; this ticket leaves both live.
|
||||
```
|
||||
test_5t_transfer_learning_runs
|
||||
test_5t_triangulation_runs
|
||||
test_5t_truthtables_runs
|
||||
test_5t_transitivity_runs
|
||||
test_5t_time_runs
|
||||
test_5t_v1_transfer_digest_unchanged # guard regression
|
||||
test_5t_fixture_accepts_carrier_metadata
|
||||
test_5t_text_runner_rejects_unsupported_nontext_carrier_cleanly
|
||||
test_5t_time_reads_memory_root_fixture
|
||||
test_5t_transitivity_rejects_non_whitelisted_relation
|
||||
test_5t_truthtables_N_le_4_only
|
||||
```
|
||||
|
||||
Stable-digest tests for each new fixture file. Remove the four
|
||||
implemented sub-batteries from any "stub returns zero"
|
||||
parametrization.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
## 7. Out of scope
|
||||
|
||||
- **Truthtables surface.** arborist doesn't currently do
|
||||
propositional evaluation; building the wrapper is real work.
|
||||
Defer if scope compresses; ship the other three first.
|
||||
- **Time integration depth.** memory_root snapshots are advisory
|
||||
(default policy off). The Time bench can still measure
|
||||
preservation across snapshots even when binding is off, but
|
||||
results may not reflect production behavior under different
|
||||
policies. Document the policy assumption in the fixture meta.
|
||||
- **Backward compatibility.** Old `bench-5t` (v1 names) stays
|
||||
alongside new `bench-5t-*` (v2 names). Risk of confusion;
|
||||
mitigate by clear naming + a `docs/bench-vocabulary.md` note
|
||||
if needed.
|
||||
- Multimodal 5T execution (depends on #000013 spatial-temporal
|
||||
substrate + π* image / audio / world canonicalizers).
|
||||
- 5S Phase 1b (ticket #000023).
|
||||
- 5F battery (ticket #000025).
|
||||
- 5R battery — Phase 2 of #000021.
|
||||
- Retiring v1 names. Once v2 is the standard, deprecate v1 in a
|
||||
follow-up; this ticket leaves both live.
|
||||
- Hidden-channel **generation**. Defensive detection only.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
## 8. Risks & mitigations
|
||||
|
||||
**Open · awaiting go/no-go.** Mid-to-large surface; biggest sub-
|
||||
battery (Time) needs memory-root integration. Recommended
|
||||
implementation sequence: rename first, then sub-batteries in the
|
||||
order Triangulation → Transitivity → Time → Truthtables.
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Vocabulary confusion (v1 vs v2) | Keep v1 transfer fixtures; add v2 transfer-learning fixtures; document Dav1DPrometheus canonical vocabulary. |
|
||||
| Truthtables scope blowup | N=2..4 only; deterministic parser; no LLM-as-judge. |
|
||||
| Transitivity overclaims | Typed relation graph; transitive-relation whitelist; negative fixtures for non-transitive relation types. |
|
||||
| Time policy ambiguity | Fixture meta includes `memory_root_binding` policy assumption; runner reports policy mismatch. |
|
||||
| Premature multimodal scope creep | Carrier metadata now; non-text carrier support later; unsupported carriers fail/skip explicitly. |
|
||||
|
||||
Closure criterion: all five 5T sub-batteries (Transfer Learning,
|
||||
Triangulation, Truthtables, Transitivity, Time) have non-stub
|
||||
runners + fixture files; `make bench-5t` runs the full set under
|
||||
v2 names; v1 `bench-5t-transfer` still passes.
|
||||
---
|
||||
|
||||
## 9. Closure criterion
|
||||
|
||||
#000024 closes when:
|
||||
|
||||
1. Dav1DPrometheus vocabulary present: Transfer Learning,
|
||||
Triangulation, Truthtables, Transitivity, Time.
|
||||
2. Legacy `transfer-v1.jsonl` digest unchanged.
|
||||
3. New `transfer-learning-v2.jsonl` lands.
|
||||
4. Triangulation runner non-stub.
|
||||
5. Transitivity runner non-stub with typed-relation whitelist.
|
||||
6. Truthtables runner non-stub with N=2..4 cap.
|
||||
7. Time runner non-stub and reads `memory_records` /
|
||||
`selfmodel_records` surfaces.
|
||||
8. All new fixtures include `carrier` / `domain` / `pi_star_ref` /
|
||||
`loss_report_refs` / `modality_notes`.
|
||||
9. Unsupported future carriers fail or skip explicitly.
|
||||
10. Make targets for each sub-battery pass.
|
||||
11. Phase 1a fixture digests unchanged.
|
||||
12. No schema, governance hash, or canonicalization-version changes.
|
||||
|
||||
---
|
||||
|
||||
## 10. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Two-phase landing recommended:
|
||||
|
||||
- **Phase 1b.1 — vocabulary alignment.** Add v2 fixture file names
|
||||
+ new sub-battery aliases. Keep v1 in place. ~150 lines.
|
||||
- **Phase 1b.2 — implement four sub-batteries.** Triangulation
|
||||
first (existing surface), Transitivity second (typed-relation
|
||||
whitelist), Time third (memory-root integrated), Truthtables
|
||||
last (new propositional wrapper). ~700-900 lines + fixtures.
|
||||
|
||||
The architectural correction:
|
||||
|
||||
> **5T evaluates reasoning transformations across states, domains,
|
||||
> evidence paths, logical cases, and time. Those transformations
|
||||
> can live in text today, but must be representable across any
|
||||
> sign-bearing or state-bearing carrier later.**
|
||||
|
||||
This makes #000024 the first ticket where v8 substrate becomes
|
||||
measurable bench territory and the path to spatial-temporal /
|
||||
multimodal / world-model AGI evaluation stays open.
|
||||
|
|
|
|||
|
|
@ -1,190 +1,376 @@
|
|||
# Ticket #000025 — 5F battery: Function, Finetuning, Falsification, Formulate, Feedback Loop
|
||||
# Ticket #000025 — 5F battery: Function · Finetuning · Falsification · Formulate · Feedback Loop
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Open the 5F battery — Dav1DPrometheus's third evaluation
|
||||
axis, missing entirely from arborist before this ticket. Each sub-
|
||||
battery measures a distinct dimension of agent quality:
|
||||
**Function** (task alignment), **Finetuning** (adaptation
|
||||
efficiency), **Falsification** (error detection + correction),
|
||||
**Formulate** (logical structuring), **Feedback Loop** (continuous
|
||||
improvement).
|
||||
**Audience:** fox + future bench authors. Once landed, 5F + 5S + 5T
|
||||
together form the complete non-embodied AGI evaluation surface
|
||||
**Revised:** 2026-05-08 (cross-modality + state-space synthesis
|
||||
folded in per review response)
|
||||
**Scope:** Open the 5F battery — Dav1DPrometheus's
|
||||
operational-quality axis, missing entirely from arborist before
|
||||
this ticket. Five sub-batteries: **Function** (task alignment),
|
||||
**Finetuning** (adaptation efficiency), **Falsification** (error
|
||||
detection + correction), **Formulate** (logical structuring),
|
||||
**Feedback Loop** (continuous improvement). 5F integrates with
|
||||
existing arborist surfaces — `providence_cache.falsification_state`,
|
||||
`audit_events`, `memory_records` (#000017, landed),
|
||||
`selfmodel_records` (#000014, landed). No new substrate work.
|
||||
**Audience:** fox + future bench authors. Once 5F lands, 5S + 5T +
|
||||
5F together form the complete non-embodied AGI evaluation surface
|
||||
Dav1DPrometheus published.
|
||||
**Hard constraint:** 5F integrates with surfaces that already exist
|
||||
in arborist — `providence_cache.falsification_state`, `audit_events`,
|
||||
`memory_records` (#000017), `selfmodel_records` (#000014). Don't
|
||||
require new substrate work to land 5F.
|
||||
**Hard constraint:** 5F is text-first in Phase 1a but the fixture
|
||||
schema is **carrier-aware**. Unsupported non-text carriers MUST
|
||||
fail or skip explicitly. No LLM-as-judge in any 5F runner. No
|
||||
schema bump, no governance hash bump. Hidden-channel work stays
|
||||
defensive (detection only, never generation).
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Fs: Function,
|
||||
Finetuning, Falsification, Formulate, Feedback Loop."* Honoring his
|
||||
legacy.
|
||||
Finetuning, Falsification, Formulate, Feedback Loop."*
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
## 1. Why 5F is the operational bridge
|
||||
|
||||
Dav1DPrometheus's framework names three orthogonal evaluation axes:
|
||||
|
||||
```text
|
||||
5Ss = linguistic precision (syntax + semantics + syllogism + synthesis + semiotics)
|
||||
5Ts = temporal/cross reasoning (transitivity + transfer learning + triangulation + truthtables + time)
|
||||
5Fs = operational quality (function + finetuning + falsification + formulate + feedback loop)
|
||||
```
|
||||
|
||||
arborist's bench harness (ticket #000021) implements 5S + 5T as
|
||||
fixture-driven batteries. **5F has no representation in arborist.**
|
||||
The original SQD whitepaper omitted it; Dav1DPrometheus's framework
|
||||
is the only source for it.
|
||||
5S evaluates linguistic / sign-structure precision.
|
||||
5T evaluates temporal / cross-reasoning / logical coverage.
|
||||
**5F evaluates whether the organism acts, adapts, falsifies,
|
||||
formulates, and integrates feedback** — the dimensions that turn a
|
||||
state-space into an *adaptive* state-space.
|
||||
|
||||
Without 5F:
|
||||
|
||||
| Sub-battery | Why arborist needs it |
|
||||
|---|---|
|
||||
| **Function** | Measures whether the system actually executes its designated task. arborist's QA layer has answer-mode bench coverage but not a clean "did the system DO what it was asked" metric. |
|
||||
| **Finetuning** | Measures speed/efficiency of adaptation. SelfModel parent→child transitions are unbenchmarked. |
|
||||
| **Falsification** | Error-detection accuracy is core to v9.8 admissibility, but `providence_cache.falsification_state` transitions are not benched. |
|
||||
| **Formulate** | Coherence of organized knowledge structures — directly the claim-lattice's job, currently only measured by per-claim verification rate, not chain-level coherence. |
|
||||
| **Feedback Loop** | Integration efficiency for incoming corrections. Memory-root branch projections (#000017) update on snapshot but speed of integration is unmeasured. |
|
||||
|
||||
### 1.1 Why this is a separate ticket from #000021
|
||||
|
||||
#000021 was scoped to the SQD whitepaper's 5S + 5T. Adding 5F
|
||||
would have changed scope mid-design. Treating 5F as its own
|
||||
ticket keeps each battery's story coherent and lets fox approve
|
||||
or punt 5F independently.
|
||||
|
||||
---
|
||||
|
||||
## 2. Sub-battery semantics (per Dav1DPrometheus)
|
||||
|
||||
### 2.1 Function
|
||||
|
||||
> *Task Alignment: Precision in executing designated tasks.
|
||||
> Operational Effectiveness: Empirical success rate in task execution.*
|
||||
|
||||
Each task: a directive (e.g., "summarize document X"), a target
|
||||
output shape (e.g., claim-lattice with N claims), and an
|
||||
evaluation predicate (does the produced output meet the shape?).
|
||||
Metric: function-pass-rate.
|
||||
|
||||
Maps to arborist surfaces: `arborist ask` invocations under
|
||||
specific `answer_mode` settings.
|
||||
|
||||
### 2.2 Finetuning
|
||||
|
||||
> *Performance Improvement: Quantitative percentage increase in
|
||||
> performance metrics post-finetuning.
|
||||
> Adaptability: Efficiency in adapting to new tasks, measured by
|
||||
> time and resources required.*
|
||||
|
||||
Each task: a parent SelfModel + a target capability claim that
|
||||
the parent fails + a child SelfModel that should pass. Metric:
|
||||
fraction of parent→child transitions where the child's measured
|
||||
metric improves over parent's.
|
||||
|
||||
Maps to arborist surfaces: `selfmodel_records` parent→child chain
|
||||
+ `selfmodel_capability_claims` measured_value tracking.
|
||||
|
||||
### 2.3 Falsification
|
||||
|
||||
> *Error Detection Accuracy: Rate of correctly identifying system
|
||||
> errors. Correction Effectiveness: Success rate in rectifying
|
||||
> identified errors.*
|
||||
|
||||
Each task: a known-bad providence record (planted) + the expected
|
||||
falsification reason. Metric: fraction of planted records the
|
||||
verifier correctly flags + fraction of repair operations that
|
||||
restore correctness.
|
||||
|
||||
Maps to arborist surfaces: `falsification_state` transitions,
|
||||
`providence_repair` audit events, `arborist.qa.repair`.
|
||||
|
||||
### 2.4 Formulate
|
||||
|
||||
> *Logical Structuring: Degree of coherence in organized
|
||||
> knowledge structures.
|
||||
> Framework Clarity: Ease of comprehension and interpretability of
|
||||
> structured knowledge.*
|
||||
|
||||
Each task: a free-text input + the expected claim-lattice
|
||||
structure (N claims, ordered, with pointer IDs). Metric: structural-
|
||||
match-rate (sorted-claim-text + pointer-ID-set agreement, NOT
|
||||
exact-string).
|
||||
|
||||
Maps to arborist surfaces: `arborist.qa.parse_claims` +
|
||||
`arborist.pi_star.claim_lattice`.
|
||||
|
||||
### 2.5 Feedback Loop
|
||||
|
||||
> *Integration Efficiency: Speed and efficacy in assimilating
|
||||
> feedback into system processes.
|
||||
> Continuous Improvement: Measurable performance enhancements
|
||||
> through iterative feedback mechanisms.*
|
||||
|
||||
Each task: a sequence of (operation, observation) pairs forming a
|
||||
feedback chain; the system should integrate observations into
|
||||
subsequent operations. Metric: integration-coverage-rate (fraction
|
||||
of observations that affect a downstream snapshot).
|
||||
|
||||
Maps to arborist surfaces: `memory_branch_summaries` updates +
|
||||
`audit_events` chain delta between snapshots.
|
||||
|
||||
---
|
||||
|
||||
## 3. Design choices
|
||||
|
||||
### 3.1 Battery vs sibling layer
|
||||
|
||||
**A. Treat 5F as a peer to 5S/5T inside `bench/batteries/`.** Same
|
||||
runner skeleton; new file `b_5f.py`.
|
||||
|
||||
**B. Treat 5F as a separate top-level harness.** New module
|
||||
`bench/quality/`. Distinct because Function/Finetuning are
|
||||
operational rather than substrate-evaluation properties.
|
||||
|
||||
**Recommendation:** A. Keeps the harness uniform; the runner CLI
|
||||
already supports arbitrary battery names. The semantic distinction
|
||||
between 5S/5T (substrate) and 5F (operational) is documented in
|
||||
the ticket but doesn't need a code-level distinction.
|
||||
|
||||
### 3.2 Fixture sourcing per sub-battery
|
||||
|
||||
| Sub-battery | Fixture source |
|
||||
|---|---|
|
||||
| Function | Hand-curated `(directive, expected-shape, evaluator)` triples. |
|
||||
| Finetuning | Generated from `selfmodel_records` chains in actual shards (or test fixtures). |
|
||||
| Falsification | Planted-error fixtures: known-bad providence records with stated falsification reason. |
|
||||
| Formulate | Hand-curated `(text, expected-claim-lattice)` pairs from existing claim-lattice tests. |
|
||||
| Feedback Loop | Multi-snapshot fixture chains; each chain tagged with expected memory-root delta semantics. |
|
||||
|
||||
### 3.3 Phase staging
|
||||
|
||||
Like 5S/5T, 5F lands in two phases:
|
||||
|
||||
- **Phase 1a (this ticket scope):** runner skeleton in `b_5f.py`,
|
||||
~10 seed fixtures per sub-battery, all sub-batteries return real
|
||||
results (no stubs). Get the surface up.
|
||||
- **Phase 1b (follow-up):** expand fixtures to 30-50 per sub-battery
|
||||
for bench-floor signal.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Module + fixtures
|
||||
|
||||
```
|
||||
bench/batteries/b_5f.py # new; Function/Finetuning/Falsification/Formulate/Feedback runners
|
||||
bench/fixtures/5f/function-v1.jsonl
|
||||
bench/fixtures/5f/finetuning-v1.jsonl
|
||||
bench/fixtures/5f/falsification-v1.jsonl
|
||||
bench/fixtures/5f/formulate-v1.jsonl
|
||||
bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||
state-space exists but may not improve.
|
||||
```
|
||||
|
||||
### 4.2 Runner registration
|
||||
With 5F:
|
||||
|
||||
```
|
||||
state-space becomes an adaptive learning organism.
|
||||
```
|
||||
|
||||
That is the operational gap 5F closes.
|
||||
|
||||
---
|
||||
|
||||
## 2. State-space / time synthesis
|
||||
|
||||
The folded-in correction (2026-05-08 review):
|
||||
|
||||
> **SQD + Merkle-AGI v7 + 5S/5T/5F + arborist together instantiate
|
||||
> an explicit discrete state-space/time in which counting,
|
||||
> mathematics, and logic become auditable operations over committed
|
||||
> state — not just text generated by a latent model.**
|
||||
|
||||
Define an arborist organism state at time `t`:
|
||||
|
||||
```
|
||||
Ω_t = (
|
||||
W_t, workspace state
|
||||
I_t, invariant objects under π*
|
||||
C_t, concept / semiotic state
|
||||
L_t, claim lattice
|
||||
MRoot_t, memory root
|
||||
SMRoot_t, SelfModel root
|
||||
PRoot_t, policy root
|
||||
BRoot_t, benchmark root
|
||||
ARoot_t, audit root
|
||||
)
|
||||
```
|
||||
|
||||
A transition is `Ω_{t+1} = T(Ω_t, Δ_t)` where `Δ_t ∈ {new evidence,
|
||||
new claim, new falsification, new benchmark result, new SelfModel
|
||||
update, new memory snapshot, new feedback observation, new repair
|
||||
operation}`. This is a discrete state-space/time — not metaphorical.
|
||||
The system has states, transitions, invariants, events, memory,
|
||||
verification, branching, rollback, selection pressure.
|
||||
|
||||
5F maps onto the transitions:
|
||||
|
||||
| Sub-battery | Transition question |
|
||||
|---|---|
|
||||
| Function | Does the transition do the intended operation? |
|
||||
| Finetuning | Does the organism improve across parent→child states? |
|
||||
| Falsification | Does the organism reject invalid states? |
|
||||
| Formulate | Does the organism convert raw input into structured state? |
|
||||
| Feedback Loop | Does observation at `t` affect state at `t+1`? |
|
||||
|
||||
**Counting** emerges over finite committed sets (claims, verifier
|
||||
passes, memory branches, feedback observations integrated, SelfModel
|
||||
capability upgrades). **Mathematics** emerges over π*-canonical
|
||||
invariants. **Logic** emerges from verifier predicates, truthtables,
|
||||
syllogisms, transitivity, falsification states, admissibility
|
||||
labels. **Time** emerges from ordered state transitions across
|
||||
MemoryRoot / SelfModel / BenchmarkRoot.
|
||||
|
||||
The break from traditional LLM stacks:
|
||||
|
||||
> **Traditional LLMs generate math/logic as behavior. This stack
|
||||
> externalizes math/logic as committed, verifier-addressable state.**
|
||||
|
||||
That is the architectural justification for 5F's existence as a
|
||||
distinct axis.
|
||||
|
||||
---
|
||||
|
||||
## 3. Carrier-aware 5F definitions
|
||||
|
||||
Common rule:
|
||||
|
||||
> A 5F task evaluates whether the organism can operate correctly
|
||||
> over a carrier, adapt over that carrier, falsify errors in that
|
||||
> carrier, formulate structure from it, or integrate feedback about
|
||||
> it.
|
||||
|
||||
Phase 1a allowed carriers:
|
||||
|
||||
```
|
||||
text · claim_lattice · providence_record ·
|
||||
selfmodel_snapshot · memory_snapshot · audit_event
|
||||
```
|
||||
|
||||
Future carriers:
|
||||
|
||||
```
|
||||
code · table · image · audio · video · sensor ·
|
||||
spatial/world states · tool_action · mixed · hidden_channel
|
||||
```
|
||||
|
||||
Unsupported carriers in v1 → `reason="unsupported_carrier"`,
|
||||
explicit fail or skip.
|
||||
|
||||
---
|
||||
|
||||
## 4. Fixture schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-<sub>-NNN",
|
||||
"battery": "5f",
|
||||
"sub_battery": "function",
|
||||
"version": "v1",
|
||||
|
||||
"carrier": "claim_lattice",
|
||||
"domain": "qa_answer",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"loss_report_refs": [],
|
||||
"modality_notes": "Phase 1a uses text/claim-lattice; future carriers may include image, code, tool traces, world state.",
|
||||
|
||||
"task": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Per-sub-battery design
|
||||
|
||||
### 5.1 Function
|
||||
|
||||
**Definition:** does the system actually perform the designated
|
||||
operation? Phase 1a: directive → expected output shape →
|
||||
deterministic predicate.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-function-001",
|
||||
"carrier": "claim_lattice",
|
||||
"domain": "qa_answer",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"directive": "Return exactly two claim-lattice claims with evidence pointers.",
|
||||
"answer_mode": "claim_lattice",
|
||||
"expected_shape": {
|
||||
"claim_count": 2,
|
||||
"pointers_required": true
|
||||
},
|
||||
"evaluator": "shape_match",
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**v1 evaluators (deterministic only):**
|
||||
|
||||
```
|
||||
shape_match · pointer_set_match · threshold_on_metric
|
||||
```
|
||||
|
||||
**Future evaluators:** `ast_patch_applies`, `image_region_detected`,
|
||||
`world_state_updated`, `tool_postcondition_met`.
|
||||
|
||||
**Metric:** `function_pass_rate = passed / total`.
|
||||
|
||||
### 5.2 Finetuning
|
||||
|
||||
**Definition:** measured improvement from parent state to child
|
||||
state after adaptation. Adaptation includes SelfModel update, policy
|
||||
update, benchmark-selected patch, π* canonicalizer update,
|
||||
memory-root update, model-profile change, actual model finetune.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-finetuning-001",
|
||||
"carrier": "selfmodel_snapshot",
|
||||
"domain": "capability_transition",
|
||||
"pi_star_ref": "pi_selfmodel_v1",
|
||||
"parent_selfmodel": "SM_PARENT",
|
||||
"child_selfmodel": "SM_CHILD",
|
||||
"target_capability": "CAP-5S-SYLLOGISM",
|
||||
"parent_measured_value": 0.45,
|
||||
"child_measured_value": 0.65,
|
||||
"expected_improvement_min": 0.05,
|
||||
"resource_budget": {
|
||||
"max_compute_ms_delta": 1000,
|
||||
"max_storage_delta_bytes": 1000000
|
||||
},
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Metrics:**
|
||||
|
||||
```
|
||||
adaptation_improvement_rate
|
||||
adaptation_efficiency = improvement_delta / capital_cost_delta
|
||||
```
|
||||
|
||||
`adaptation_efficiency` hooks into the capital ledger (#000020,
|
||||
landed) and prepares fitness-cost weighing for v8 fork choice
|
||||
(#000012).
|
||||
|
||||
### 5.3 Falsification
|
||||
|
||||
**Definition:** does the system detect and correct errors?
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-falsification-001",
|
||||
"carrier": "providence_record",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"record": "planted_bad_providence_record",
|
||||
"expected_reason": "WARRANT_MISSING",
|
||||
"verifier_method_root": "...",
|
||||
"repair_expected": true,
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Metrics:** `error_detection_rate`, `repair_success_rate`.
|
||||
|
||||
**Verifier-method-root tag:** every Falsification fixture pins the
|
||||
`verifier_method_root` it was authored against. If the verifier
|
||||
shape changes upstream, the bench reports a *mismatch warning*
|
||||
instead of treating the fixture as a false failure.
|
||||
|
||||
**Hidden / alternate-media safety framing:** future carriers
|
||||
(`carrier: "image"` / `"audio"` / `"hidden_channel"`) enable
|
||||
defensive Falsification tasks — *detect unexpected carrier*,
|
||||
*flag projection-loss risk*, *flag hidden-channel suspicion*,
|
||||
*reject unsupported semantics*. Never generation, never
|
||||
concealment.
|
||||
|
||||
### 5.4 Formulate
|
||||
|
||||
**Definition:** turn unstructured input into coherent, interpretable
|
||||
structure.
|
||||
|
||||
Phase 1a: free text → claim lattice.
|
||||
|
||||
Future: image → scene graph; table → relational model; code →
|
||||
AST/dependency-graph; sensor stream → event graph; world state →
|
||||
object-relation graph; mixed media → cross-modal claim lattice.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-formulate-001",
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"input_text": "Document sentence with two claims and two pointers.",
|
||||
"expected_lattice": {
|
||||
"claim_count": 2,
|
||||
"ordered": true,
|
||||
"pointer_id_sets": [["E1"], ["E2"]]
|
||||
},
|
||||
"match_policy": {
|
||||
"claim_text": "sorted_approx",
|
||||
"pointer_ids": "exact_set",
|
||||
"order": "required"
|
||||
},
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Metric:** `structural_match_rate`.
|
||||
|
||||
**Match policy:** structural (sorted-approx claim text +
|
||||
exact-set pointer IDs + required order), NOT exact-string. Avoids
|
||||
brittleness without sacrificing determinism.
|
||||
|
||||
### 5.5 Feedback Loop
|
||||
|
||||
**Definition:** do observations at `t` change downstream
|
||||
behavior/state at `t+1`?
|
||||
|
||||
This is the most important 5F component for recursive
|
||||
self-improvement. Without Feedback Loop, the state-space exists
|
||||
but does not learn.
|
||||
|
||||
**Fixture shape:**
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "5f-feedback-loop-001",
|
||||
"carrier": "memory_snapshot",
|
||||
"domain": "memory_root",
|
||||
"pi_star_ref": "pi_memory_v1",
|
||||
"chain": [
|
||||
{"operation": "snapshot", "observation": "claim X is stale"},
|
||||
{"operation": "update_memory", "observation": "memory branch marks X stale"},
|
||||
{"operation": "snapshot", "expected_delta": "X appears in stale set"}
|
||||
],
|
||||
"expected": "pass"
|
||||
}
|
||||
```
|
||||
|
||||
**Metrics:**
|
||||
|
||||
```
|
||||
integration_coverage_rate
|
||||
feedback_latency
|
||||
downstream_effect_rate
|
||||
feedback_efficiency = downstream_effect_count / capital_cost_delta
|
||||
```
|
||||
|
||||
`feedback_efficiency` again hooks the capital ledger (#000020).
|
||||
|
||||
**Phase 1a uses existing surfaces only:** `memory_records`,
|
||||
`memory_branch_summaries`, `audit_events`. No new substrate work.
|
||||
|
||||
---
|
||||
|
||||
## 6. Required file changes
|
||||
|
||||
### 6.1 Module + fixtures
|
||||
|
||||
```
|
||||
bench/batteries/b_5f.py # new
|
||||
bench/fixtures/5f/function-v1.jsonl # ~10 seed (Phase 1a)
|
||||
bench/fixtures/5f/finetuning-v1.jsonl # ~10 seed
|
||||
bench/fixtures/5f/falsification-v1.jsonl # ~10 seed
|
||||
bench/fixtures/5f/formulate-v1.jsonl # ~10 seed
|
||||
bench/fixtures/5f/feedback-loop-v1.jsonl # ~10 seed
|
||||
```
|
||||
|
||||
Each fixture file carries the standard `_meta` head with
|
||||
`battery="5f"`.
|
||||
|
||||
### 6.2 Runner registration
|
||||
|
||||
In `bench/batteries/runner.py`:
|
||||
|
||||
|
|
@ -196,79 +382,147 @@ _BATTERIES = {
|
|||
}
|
||||
```
|
||||
|
||||
Existing `--all` mode picks up the new fixtures automatically once
|
||||
they're in `_DEFAULT_FIXTURES`.
|
||||
|
||||
### 4.3 Makefile additions
|
||||
### 6.3 Make targets
|
||||
|
||||
```makefile
|
||||
bench-5f: bootstrap ## 5F battery (Function/Finetuning/Falsification/Formulate/Feedback Loop)
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub function --fixtures bench/fixtures/5f/function-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub finetuning --fixtures bench/fixtures/5f/finetuning-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub falsification --fixtures bench/fixtures/5f/falsification-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub formulate --fixtures bench/fixtures/5f/formulate-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop --fixtures bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||
bench-5f-function:
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub function \
|
||||
--fixtures bench/fixtures/5f/function-v1.jsonl
|
||||
|
||||
bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite
|
||||
bench-5f-finetuning:
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub finetuning \
|
||||
--fixtures bench/fixtures/5f/finetuning-v1.jsonl
|
||||
|
||||
bench-5f-falsification:
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub falsification \
|
||||
--fixtures bench/fixtures/5f/falsification-v1.jsonl
|
||||
|
||||
bench-5f-formulate:
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub formulate \
|
||||
--fixtures bench/fixtures/5f/formulate-v1.jsonl
|
||||
|
||||
bench-5f-feedback-loop:
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop \
|
||||
--fixtures bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||
|
||||
bench-5f: bench-5f-function bench-5f-finetuning \
|
||||
bench-5f-falsification bench-5f-formulate \
|
||||
bench-5f-feedback-loop
|
||||
|
||||
bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite
|
||||
```
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- `tests/test_bench_batteries.py` extension: positive coverage
|
||||
for each new sub-battery.
|
||||
- Determinism tests: each runner produces identical
|
||||
`BatteryResult.fixture_digest` and `runtime_digest` across
|
||||
back-to-back runs.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New module `bench/batteries/b_5f.py`.
|
||||
- 5 new fixture files.
|
||||
- New Makefile targets.
|
||||
- No schema changes; no policy hash changes.
|
||||
`make bench-5s5t5f` becomes the canonical Dav1DPrometheus
|
||||
non-embodied AGI fitness surface.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
## 7. Tests
|
||||
|
||||
- Phase 1b fixture expansion (target 30-50 per sub-battery). Phase
|
||||
1a's seed set lands first.
|
||||
- Cross-battery composition (e.g., "score this agent across
|
||||
5S+5T+5F"). Aggregation is its own concern.
|
||||
- Calibration of metric thresholds. Phase 1a measures; thresholds
|
||||
for v8 selection acceptance come later.
|
||||
- Auto-grading via LLM. All 5F evaluators are deterministic
|
||||
predicates; LLM-as-judge is future work.
|
||||
```
|
||||
test_5f_function_runs
|
||||
test_5f_finetuning_runs
|
||||
test_5f_falsification_runs
|
||||
test_5f_formulate_runs
|
||||
test_5f_feedback_loop_runs
|
||||
test_5f_fixture_digest_stable
|
||||
test_5f_runtime_digest_stable
|
||||
test_5f_fixture_accepts_carrier_metadata
|
||||
test_5f_unsupported_future_carrier_fails_cleanly
|
||||
```
|
||||
|
||||
Sub-battery-specific:
|
||||
|
||||
```
|
||||
test_5f_function_shape_match_evaluator
|
||||
test_5f_function_pointer_set_match_evaluator
|
||||
test_5f_function_threshold_on_metric_evaluator
|
||||
test_5f_finetuning_requires_parent_and_child_selfmodel
|
||||
test_5f_falsification_verifier_method_root_mismatch_warns
|
||||
test_5f_formulate_pointer_id_set_exact_match
|
||||
test_5f_feedback_loop_observation_affects_downstream_snapshot
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
## 8. Out of scope
|
||||
|
||||
- **Function evaluator design.** "Did the system execute the
|
||||
designated task" requires a per-task predicate. v1 ships a
|
||||
small set of canonical evaluator types (shape-match,
|
||||
pointer-set-match, threshold-on-metric). New evaluator types
|
||||
land in follow-ups.
|
||||
- **Finetuning needs SelfModel chains.** Realistic fixtures
|
||||
require real shard data with parent→child SelfModel transitions.
|
||||
Phase 1a uses synthetic chains generated in-test; Phase 1b
|
||||
pulls from production shards.
|
||||
- **Falsification fixtures are fragile.** A planted-error fixture
|
||||
bound to a specific verifier signature may stop catching the
|
||||
error if the verifier changes shape. Mitigation: tag each
|
||||
Falsification fixture with the `verifier_method_root` it was
|
||||
authored against; bench reports flag mismatches.
|
||||
- Phase 1b fixture expansion (target 30-50 per sub-battery).
|
||||
- Cross-battery composite scoring (5S + 5T + 5F aggregation is its
|
||||
own concern).
|
||||
- Calibration of metric thresholds for v8 selection acceptance
|
||||
(depends on #000012 v8 paper).
|
||||
- LLM-as-judge for any 5F task (forbidden by hard constraint).
|
||||
- Multimodal Phase 1a (text/claim-lattice/memory only; multimodal
|
||||
comes after #000013 + #000015 follow-ups land).
|
||||
- Hidden-channel **generation**. Defensive detection only, in
|
||||
Phase 1b+.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
## 9. Risks & mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| Function evaluator vague | v1 allows only named deterministic evaluator types: `shape_match`, `pointer_set_match`, `threshold_on_metric`. Future evaluators land in follow-ups. |
|
||||
| Finetuning fakes improvement | Require parent + child SelfModel roots; require measured_value pre/post; require improvement threshold; record capital cost. |
|
||||
| Falsification fixtures stale | Tag every fixture with `verifier_method_root`. Bench reports mismatch instead of false failure. |
|
||||
| Formulate exact-string brittle | Structural match only: claim count + order + pointer-ID-set + claim-text approximate/sorted. |
|
||||
| Feedback Loop needs substrate work | Use existing `memory_records`, `memory_branch_summaries`, `audit_events` surfaces. Phase 1a explicitly forbids new substrate. |
|
||||
| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers fail/skip explicitly. |
|
||||
|
||||
---
|
||||
|
||||
## 10. Closure criterion
|
||||
|
||||
### Phase 1a closes when:
|
||||
|
||||
1. `bench/batteries/b_5f.py` lands.
|
||||
2. All five sub-battery runners are non-stub.
|
||||
3. Each sub-battery has ≥10 deterministic seed fixtures.
|
||||
4. Fixture metadata includes `carrier` / `domain` / `pi_star_ref`
|
||||
/ `loss_report_refs` / `modality_notes` fields.
|
||||
5. `make bench-5f` passes end-to-end.
|
||||
6. `make bench-5s5t5f` exists and runs the complete Dav1DPrometheus
|
||||
suite.
|
||||
7. Runtime + fixture digests are deterministic across repeated
|
||||
runs.
|
||||
8. No schema, governance hash, or canonicalization-version changes.
|
||||
9. No LLM-as-judge anywhere in 5F runners.
|
||||
|
||||
### Phase 1b keeps the ticket in-progress until:
|
||||
|
||||
10. 30-50 fixtures per sub-battery.
|
||||
11. Real shard-based Finetuning chains (replacing synthetic
|
||||
parent→child SelfModel pairs from Phase 1a).
|
||||
12. Expanded Falsification planted-error suite covering every
|
||||
documented failure-motif tag.
|
||||
13. Feedback Loop latency / efficiency metrics calibrated against
|
||||
real workload data.
|
||||
14. Threshold calibration for v8 selection acceptance handed off
|
||||
to ticket #000012.
|
||||
|
||||
---
|
||||
|
||||
## 11. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Larger surface than #000023 / #000024
|
||||
because 5F has five sub-batteries vs three remaining in 5S and
|
||||
four in 5T. Recommended sequencing: ship Phase 1a in one push
|
||||
(seed-fixture per sub-battery, ~10 each), then Phase 1b expands.
|
||||
(five sub-batteries vs three / four remaining). Recommended
|
||||
sequencing: ship Phase 1a in one push (seed fixtures + non-stub
|
||||
runners), Phase 1b expands.
|
||||
|
||||
Closure criterion (Phase 1a): `bench/batteries/b_5f.py` lands
|
||||
with non-stub runners for all five sub-batteries; seed fixtures
|
||||
+ Makefile targets pass `make bench-5f` end-to-end. Phase 1b
|
||||
keeps the ticket open in-progress until fixtures are expanded.
|
||||
The synthesis statement that justifies 5F's existence as a distinct
|
||||
axis:
|
||||
|
||||
> **SQD gives the invariant projection and verifier model.
|
||||
> Merkle-AGI v7 gives multimodal committed DAG structure. 5S gives
|
||||
> sign/meaning/derivation fitness. 5T gives transfer, logic,
|
||||
> triangulation, and time. 5F gives operation, adaptation,
|
||||
> falsification, formulation, and feedback. Together they
|
||||
> instantiate an explicit discrete state-space/time in which
|
||||
> counting, mathematics, and logic become auditable operations over
|
||||
> committed state — not just text generated by a latent model.**
|
||||
|
||||
5F is the operational layer that turns the substrate into an
|
||||
adaptive learning organism. Without it, the state-space exists
|
||||
but does not improve.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue