docs/tickets: rewrite #000023/#000024/#000025 with cross-modality discipline
Per the three review responses (~/Downloads/RESPONSE_*) folded in 2026-05-08, the 5S / 5T / 5F tickets are corrected from "text-only with future hooks" to "carrier-aware design from day one." Phase 1 implementation stays text / claim-lattice / memory-root only, but the fixture schema MUST accommodate future visual / world / code / audio / sensor / hidden-channel-detection carriers without re-authoring. Common corrections across all three tickets: - Mandatory fixture metadata: carrier, domain, pi_star_ref, loss_report_refs, modality_notes. - Unsupported carriers MUST fail or skip explicitly with reason="unsupported_carrier" — never silently accepted. - No LLM-as-judge in any runner. - Hidden-channel work is defensive only (detection / flagging), never generation or concealment. Per-ticket headlines: #000023 — 5S Syntax / Semantics / Semiotics defined as carrier-general operations over sign-bearing representations. Semiotics gets the biggest correction: visual symbols, layout, metadata, encoded sign systems are valid carriers (Phase 1 still text-only). Synonym source policy: concept_relations.relation_kind='synonym' only for v1 positives. #000024 — 5T Vocabulary alignment with Dav1DPrometheus authoritative wording (Transfer→Transfer Learning, Truth→Truthtables, Timing→Time). Transitivity gets a typed-relation whitelist (implies, subset_of, ancestor_of, before, less_than) — not all edges transitive. Truthtables capped at N=2..4 to avoid combinatorial blowup. Time is the first sub-battery where v8 substrate (memory_root #000017 + selfmodel #000014) becomes a measurable bench target. #000025 — 5F New axis. Function/Finetuning/Falsification/Formulate/Feedback Loop. Folds in the state-space synthesis: SQD + v7 + 5S/5T/5F + arborist together instantiate a discrete state-space/time Ω_t = (W, I, C, L, MRoot, SMRoot, PRoot, BRoot, ARoot) with Ω_{t+1} = T(Ω_t, Δ_t). Counting / mathematics / logic / time emerge as auditable operations over committed state, not text from a latent model. adaptation_efficiency and feedback_efficiency metrics hook into the capital ledger (#000020) so v8 fork choice has cost-aware fitness signals. Falsification fixtures tagged with verifier_method_root so verifier shape changes warn rather than false-fail. All three tickets remain "open · awaiting go/no-go" — design-only. Implementation tickets land in follow-up commits when fox approves the corrected scope. Source: Legally Unprecedented Dav1DPrometheus (BasementAGI host, Where The mAGIc Happens). Honoring his framework.
This commit is contained in:
parent
02c7e41ef8
commit
fbd99a8d76
3 changed files with 1185 additions and 572 deletions
|
|
@ -1,230 +1,392 @@
|
||||||
# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics
|
# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics (carrier-aware)
|
||||||
|
|
||||||
**Status:** open · awaiting go/no-go
|
**Status:** open · awaiting go/no-go
|
||||||
**Opened:** 2026-05-07
|
**Opened:** 2026-05-07
|
||||||
**Scope:** Complete the 5S battery's three remaining sub-batteries —
|
**Revised:** 2026-05-08 (cross-modality correction folded in per
|
||||||
Syllogism, Synthesis, Semiotics — that landed as zero-task stubs in
|
review response)
|
||||||
ticket #000021 Phase 1a. Each sub-battery gets a real fixture set
|
**Scope:** Replace the three zero-task stubs in 5S
|
||||||
(seed → expand) and a runner that exercises arborist's actual
|
(Syllogism / Synthesis / Semiotics) with deterministic Phase-1b
|
||||||
surface (claim-lattice parser, retrieval pipeline, label-swap
|
runners + fixture sets. Implementation stays text / claim-lattice /
|
||||||
behavior).
|
prose only; **the fixture schema is carrier-aware** so future
|
||||||
**Audience:** fox + future bench authors + downstream v8 / v7-W work
|
phases can attach visual, world-model, code, audio, sensor, and
|
||||||
that needs concrete fitness targets across all 5S dimensions.
|
hidden-channel-detection carriers without re-authoring the
|
||||||
**Hard constraint:** fixtures land deterministically (committed
|
fixture format.
|
||||||
JSONL with version pin). Bench-maxing discipline (5pp signal floor,
|
**Audience:** fox + future bench authors + downstream v7-W /
|
||||||
n≥3) applies. Each sub-battery is independently runnable via
|
multimodal work that needs 5S evaluation across non-text carriers.
|
||||||
`make bench-5s-<sub>`. Adding sub-batteries does NOT alter Phase 1a
|
**Hard constraint:** Phase 1a fixture digests stay pinned (Syntax
|
||||||
behavior — Syntax + Semantics keep their existing fixture digests.
|
+ Semantics keep their `v1` digests). No schema bump, no
|
||||||
|
governance hash bump, no canonicalization-version bump. Unsupported
|
||||||
|
non-text carriers MUST fail or skip explicitly — never silently
|
||||||
|
accepted. No LLM-as-judge in any 5S runner.
|
||||||
|
|
||||||
**Source:** Dav1DPrometheus (BasementAGI host) named the 5Ss in his
|
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ss: Syntax,
|
||||||
authoritative framework; arborist honors the original wording.
|
Semantics, Syllogism, Synthesis, Semiotics."*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Problem statement
|
## 1. Corrected purpose
|
||||||
|
|
||||||
Ticket #000021 Phase 1a shipped runners + seed fixtures for 5S
|
Two coupled goals:
|
||||||
Syntax (`parse-pass`) and 5S Semantics (`equivalence`). The other
|
|
||||||
three sub-batteries — Syllogism, Synthesis, Semiotics — landed as
|
|
||||||
stubs that return zero-task `BatteryResult` objects so the full
|
|
||||||
suite at least runs. Stubs are visible in `bench/batteries/b_5s.py`.
|
|
||||||
|
|
||||||
Without real implementations:
|
1. **Complete the 5S battery** — three stubs become real runners
|
||||||
|
with ≥30 deterministic fixtures each.
|
||||||
|
2. **Make 5S carrier-general** — Syntax, Semantics, Semiotics,
|
||||||
|
Syllogism, Synthesis are not prose-only. They are tests over
|
||||||
|
*sign-bearing representations*. Text is only the first carrier.
|
||||||
|
|
||||||
| Gap | Failure mode |
|
The architectural correction (folded in 2026-05-08):
|
||||||
|
|
||||||
|
> **Syntax hidden in visual or alternate media is still syntax.
|
||||||
|
> Meaning hidden in non-text media is still semantics. A sign
|
||||||
|
> encoded in a non-text carrier is still semiotics.**
|
||||||
|
|
||||||
|
#000023 lands text fixtures + text runners now. The fixture
|
||||||
|
**schema** must accommodate future cross-carrier work without
|
||||||
|
re-authoring.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Carrier-aware 5S definitions
|
||||||
|
|
||||||
|
| Sub-battery | Definition (carrier-general) |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **Syllogism** | No way to bench arborist's deductive-reasoning surface. v8 selection + SelfModel capability claims have no measurable target for "stepwise entailment validity." |
|
| **Syntax** | Formal arrangement rules of a carrier. Text: tokens, parse, markup. Code: AST, imports, indentation. Tables: rows/columns/headers. Images: layout, regions, visual grouping. World: object relations, spatial containment, event ordering. Audio: phoneme/order structure, timing. |
|
||||||
| **Synthesis** | No bench for "given a goal + a fact set, can the system produce a passing derivation?" — directly the claim-lattice + retrieval pipeline's output, which today only gets ad-hoc QA bench coverage. |
|
| **Semantics** | Invariant meaning preserved across representation changes. Text: synonym pairs, entity equivalence. Image: a stop sign and the text "STOP" carry equivalent traffic-control meaning. Chart: an upward trend and "X increases over time" carry equivalent data semantics. World: "A inside B" represented visually, symbolically, or as coordinates. |
|
||||||
| **Semiotics** | Label-swap invariance under controlled symbol changes is unmeasured. Drift in proper-noun handling, code-name conventions, or alphabet shifts goes undetected. |
|
| **Syllogism** | Stepwise valid deduction, given premises in any carrier. Text+table: "all rows with status=active have flag=true; row X has status=active; therefore X has flag=true." Image+rule: "A is inside B; rule says objects in B have property P; therefore A has P." World: "A left of B; B left of C; therefore A left of C." |
|
||||||
|
| **Synthesis** | Assembling multiple warranted elements into a coherent derivation. Text+chart: derive a claim only when textual and chart evidence agree. Image+OCR+metadata: synthesize while flagging carrier conflict. World-state: synthesize a temporal claim from multiple state observations. |
|
||||||
|
| **Semiotics** | Interpretation of signs, symbols, labels, references, frames, conventions, aliases, sign systems. Text: AMD↔"Advanced Micro Devices", Apple-fruit vs Apple-company, Orwell-frame vs literal-geography. Visual: red octagon = stop, skull icon = hazard, watermark = provenance, QR-pattern = encoded sign system, diagram-arrow = causality/flow/dependency. Future: visual steganographic carriers where surface differs from hidden encoding. |
|
||||||
|
|
||||||
### 1.1 Sub-battery semantics (per ticket #000021 §4.2)
|
Phase 1b implements only **text / claim-lattice / prose** carriers.
|
||||||
|
Future-carrier hooks land via fixture metadata (§3) that the v1
|
||||||
- **Syllogism:** each task is a proof chain + a step index +
|
runner ignores or rejects-cleanly.
|
||||||
expected step-valid pass/fail. Metric: step-validity-rate.
|
|
||||||
- **Synthesis:** each task is a goal description + a fact set +
|
|
||||||
expected derivation pass/fail. Metric: derivation-pass-rate.
|
|
||||||
- **Semiotics:** each task is a controlled label-swap + invariant
|
|
||||||
expected. Metric: invariance-under-swap.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Design choices
|
## 3. Fixture schema (carrier-aware)
|
||||||
|
|
||||||
### 2.1 Fixture sourcing for Syllogism
|
Common JSONL header per task:
|
||||||
|
|
||||||
**A. Hand-curated proof chains drawn from existing claim-lattice tests.**
|
```json
|
||||||
arborist's `tests/test_claim_lattice.py` and `tests/test_directives.py`
|
{
|
||||||
already contain stepwise-claim examples. Promoting them to
|
"id": "5s-<sub>-NNN",
|
||||||
fixtures gets us 30-50 entries cheaply.
|
"battery": "5s",
|
||||||
|
"sub_battery": "syllogism",
|
||||||
|
"version": "v1",
|
||||||
|
|
||||||
**B. Synthetic generators (template-based syllogisms).** Cheap;
|
"carrier": "text",
|
||||||
templates need to be committed for reproducibility.
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"loss_report_refs": [],
|
||||||
|
"modality_notes": "Phase 1b: text-only; future carriers may include image/code/audio/world/hidden-channel.",
|
||||||
|
|
||||||
**Recommendation:** A first; add B in Phase 2 once the harness
|
"task": { ... }
|
||||||
stabilizes.
|
}
|
||||||
|
|
||||||
### 2.2 Fixture sourcing for Synthesis
|
|
||||||
|
|
||||||
**A. Ablate from existing QA fixtures.** Take questions from
|
|
||||||
`bench/qa_sweep.py` corpora; record which sources retrieve as the
|
|
||||||
fact set; mark expected derivation status.
|
|
||||||
|
|
||||||
**B. Compose from claim-lattice docs.** Use docs already in shards;
|
|
||||||
build (goal, fact_set, derivation_expected) tuples by stripping
|
|
||||||
claims and asking whether they should re-derive.
|
|
||||||
|
|
||||||
**Recommendation:** A — anchors against existing QA bench so
|
|
||||||
Synthesis-rate trends correlate with overall QA rate.
|
|
||||||
|
|
||||||
### 2.3 Fixture sourcing for Semiotics
|
|
||||||
|
|
||||||
**A. Hand-curated label-swap pairs.** Examples: `"AMD"↔"Advanced Micro
|
|
||||||
Devices"`, `"NYC"↔"New York City"`, code-name swaps inside
|
|
||||||
identical text.
|
|
||||||
|
|
||||||
**B. Generated swaps via the concept-relations table.** arborist's
|
|
||||||
`concept_relations` already has synonym pairs from corpus
|
|
||||||
extraction; promote them as semiotic swap fixtures.
|
|
||||||
|
|
||||||
**Recommendation:** B — leverages existing per-shard data, makes
|
|
||||||
fixtures regenerable from the corpus rather than hand-pinned.
|
|
||||||
|
|
||||||
### 2.4 Runner integration
|
|
||||||
|
|
||||||
Each sub-battery is a function in `bench/batteries/b_5s.py`
|
|
||||||
matching the existing `run_syntax`/`run_semantics` signature.
|
|
||||||
Replace the stub `_stub_result` returns with real implementations.
|
|
||||||
Existing test in `tests/test_bench_batteries.py
|
|
||||||
::test_stub_sub_batteries_return_zero` will need to flip to
|
|
||||||
non-zero assertions once these land.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Recommendation
|
|
||||||
|
|
||||||
Land in this order:
|
|
||||||
|
|
||||||
1. **Syllogism** first — purely lexical (claim-lattice traversal),
|
|
||||||
no shard reads, fastest implementation.
|
|
||||||
2. **Semiotics** second — depends on `concept_relations` lookup
|
|
||||||
but no LLM call; deterministic.
|
|
||||||
3. **Synthesis** third — requires shard reads + retrieval pipeline;
|
|
||||||
slowest, biggest surface, highest variance. Defer until
|
|
||||||
Syllogism + Semiotics stabilize the harness.
|
|
||||||
|
|
||||||
Phase 1b closure = all three sub-batteries shipping with at minimum
|
|
||||||
**30 deterministic fixtures each** and runners that integrate
|
|
||||||
cleanly into `make bench-5s`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Implementation sketch
|
|
||||||
|
|
||||||
### 4.1 Fixture files
|
|
||||||
|
|
||||||
```
|
|
||||||
bench/fixtures/5s/syllogism-v1.jsonl # 30+ tasks
|
|
||||||
bench/fixtures/5s/synthesis-v1.jsonl # 30+ tasks
|
|
||||||
bench/fixtures/5s/semiotics-v1.jsonl # 30+ tasks
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Each fixture file carries the standard `_meta` head:
|
Field semantics:
|
||||||
|
|
||||||
|
| Field | Phase 1b allowed values | Future values |
|
||||||
|
|---|---|---|
|
||||||
|
| `carrier` | `text`, `claim_lattice`, `prose` | `code`, `table`, `html`, `image`, `audio`, `layout`, `world`, `mixed`, `hidden_channel` |
|
||||||
|
| `domain` | `prose`, `wikitext`, `claim_lattice` | `python_ast`, `chart`, `spatial_state`, `propositional_logic`, etc. |
|
||||||
|
| `pi_star_ref` | `wikitext-base@v1`, `claim-lattice@v1` | future π* registry keys |
|
||||||
|
| `loss_report_refs` | `[]` | optional projection-loss record refs (#000022) |
|
||||||
|
|
||||||
|
Unsupported `carrier` / `domain` values → runner fails task
|
||||||
|
explicitly with `reason="unsupported_carrier"` or skips with same
|
||||||
|
reason logged. Never silently accepted.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Per-sub-battery implementation
|
||||||
|
|
||||||
|
### 4.1 Syllogism
|
||||||
|
|
||||||
|
**Metric:** `step_validity_rate = valid_steps / total_steps`.
|
||||||
|
|
||||||
|
**Positive fixture:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5s-syllogism-001",
|
||||||
|
"carrier": "text",
|
||||||
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"premises": [
|
||||||
|
{"claim": "All mammals are animals.", "evidence_ids": ["E1"]},
|
||||||
|
{"claim": "All dogs are mammals.", "evidence_ids": ["E2"]}
|
||||||
|
],
|
||||||
|
"candidate_step": {
|
||||||
|
"claim": "All dogs are animals.",
|
||||||
|
"uses": ["E1", "E2"]
|
||||||
|
},
|
||||||
|
"rule": "categorical_transitivity",
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Negative fixture:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5s-syllogism-NEG-001",
|
||||||
|
"premises": [
|
||||||
|
{"claim": "All dogs are animals.", "evidence_ids": ["E1"]},
|
||||||
|
{"claim": "All cats are animals.", "evidence_ids": ["E2"]}
|
||||||
|
],
|
||||||
|
"candidate_step": {
|
||||||
|
"claim": "All animals are dogs.",
|
||||||
|
"uses": ["E1", "E2"]
|
||||||
|
},
|
||||||
|
"rule": "invalid_converse",
|
||||||
|
"expected": "fail"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Runner behavior:** load fixture → parse premises into
|
||||||
|
claim-lattice → parse candidate step → check referenced prior
|
||||||
|
claims exist → check step is licensed by declared rule → return
|
||||||
|
pass/fail. Deterministic, no LLM judge.
|
||||||
|
|
||||||
|
### 4.2 Synthesis
|
||||||
|
|
||||||
|
**Metric:** `derivation_pass_rate = passed / total`.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5s-synthesis-001",
|
||||||
|
"carrier": "text",
|
||||||
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"goal": "Derive whether the source supports the release-date claim.",
|
||||||
|
"fact_set": [
|
||||||
|
{"id": "F1", "text": "Back to the Future was released on July 3, 1985."},
|
||||||
|
{"id": "F2", "text": "The film was directed by Robert Zemeckis."}
|
||||||
|
],
|
||||||
|
"expected_derivation": [
|
||||||
|
{"claim": "Back to the Future was released on July 3, 1985.", "uses": ["F1"]}
|
||||||
|
],
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Future-carrier hook:** fact-set entries may later include
|
||||||
|
non-text carriers (`carrier: "chart"`, `carrier: "image"`). v1
|
||||||
|
runner rejects them as unsupported.
|
||||||
|
|
||||||
|
**Variance control:** if a Synthesis task uses arborist's retrieval
|
||||||
|
pipeline, the fixture meta MUST pin `shard_set`, `retrieval_mode`,
|
||||||
|
and `expected_fact_ids`. Bench-maxing 5pp signal floor handles
|
||||||
|
residual noise; pinned fact IDs eliminate the largest variance
|
||||||
|
source.
|
||||||
|
|
||||||
|
### 4.3 Semiotics
|
||||||
|
|
||||||
|
**Metric:** `invariance_under_swap = invariant_passes / total`.
|
||||||
|
|
||||||
|
**Synonym-swap fixture (positive):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5s-semiotics-001",
|
||||||
|
"carrier": "text",
|
||||||
|
"domain": "prose",
|
||||||
|
"pi_star_ref": "wikitext-base@v1",
|
||||||
|
"source_text": "AMD released a new GPU.",
|
||||||
|
"swap": {
|
||||||
|
"from": "AMD",
|
||||||
|
"to": "Advanced Micro Devices",
|
||||||
|
"relation_kind": "synonym"
|
||||||
|
},
|
||||||
|
"invariant_fields": ["entity", "predicate", "object"],
|
||||||
|
"expected_invariant": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Reference-frame fixture:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5s-semiotics-002",
|
||||||
|
"source_text": "Oceania has always been at war with Eastasia.",
|
||||||
|
"frame": "orwell_1984",
|
||||||
|
"literal_frame_expected": false,
|
||||||
|
"reference_frame_expected": true,
|
||||||
|
"expected_invariant": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Future-carrier hook (documented in fixture, not executed in v1):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"future_carrier_equivalent": {
|
||||||
|
"carrier": "image",
|
||||||
|
"description": "symbolic sign or visual glyph equivalent may later replace text label"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Hidden-channel safety framing.** Phase 1b does NOT generate,
|
||||||
|
hide, or recover covert messages. Future hidden-channel fixtures
|
||||||
|
MUST be framed as **defensive detection / flagging** only:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"carrier": "image",
|
||||||
|
"sub_battery": "semiotics",
|
||||||
|
"hidden_channel_expected": true,
|
||||||
|
"task_kind": "detect_unexpected_sign_channel",
|
||||||
|
"expected": "flag"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This keeps the benchmark defensive. The capability tested is
|
||||||
|
"can the system detect a suspicious carrier" — not "can the
|
||||||
|
system encode a covert message."
|
||||||
|
|
||||||
|
**Source policy for v1 fixtures:** Semiotics fixtures use
|
||||||
|
`concept_relations.relation_kind = 'synonym'` only. Negative
|
||||||
|
fixtures pair with non-equivalent swaps explicitly tagged.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Required file changes
|
||||||
|
|
||||||
|
### 5.1 New fixture files
|
||||||
|
|
||||||
|
```
|
||||||
|
bench/fixtures/5s/syllogism-v1.jsonl # ≥30 deterministic tasks
|
||||||
|
bench/fixtures/5s/synthesis-v1.jsonl # ≥30 deterministic tasks
|
||||||
|
bench/fixtures/5s/semiotics-v1.jsonl # ≥30 deterministic tasks
|
||||||
|
```
|
||||||
|
|
||||||
|
Each carries the standard `_meta` head:
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{"_meta":{"battery":"5s","sub_battery":"syllogism","version":"v1","task_count":30}}
|
{"_meta":{"battery":"5s","sub_battery":"syllogism","version":"v1","task_count":30}}
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4.2 Runner additions
|
Phase 1a fixture digests (`syntax-v1.jsonl`, `semantics-v1.jsonl`)
|
||||||
|
stay pinned.
|
||||||
|
|
||||||
In `bench/batteries/b_5s.py`, replace stub bodies:
|
### 5.2 Runner functions
|
||||||
|
|
||||||
```python
|
In `bench/batteries/b_5s.py`, replace stub bodies of
|
||||||
def run_syllogism(fixtures_path: Path) -> BatteryResult:
|
`run_syllogism`, `run_synthesis`, `run_semiotics`. Existing
|
||||||
# for each task: parse claim chain, walk step_index, assert
|
`run_syntax`, `run_semantics`, `_runtime_digest` stay untouched.
|
||||||
# the claim references valid evidence + the prior step entails
|
|
||||||
# this one under claim-lattice semantics. Metric:
|
|
||||||
# step_validity_rate = pass_count / (pass + fail).
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Same shape for `run_synthesis`, `run_semiotics`. Existing
|
### 5.3 Make targets
|
||||||
`_runtime_digest` helper covers the runtime fingerprint.
|
|
||||||
|
|
||||||
### 4.3 Makefile additions
|
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
bench-5s-syllogism: bootstrap
|
bench-5s-syllogism: ## 5S Syllogism (step-validity-rate)
|
||||||
$(PY) -m bench.batteries.runner --battery 5s --sub syllogism \
|
$(PY) -m bench.batteries.runner --battery 5s --sub syllogism \
|
||||||
--fixtures bench/fixtures/5s/syllogism-v1.jsonl
|
--fixtures bench/fixtures/5s/syllogism-v1.jsonl
|
||||||
|
|
||||||
bench-5s-synthesis: bootstrap
|
bench-5s-synthesis: ## 5S Synthesis (derivation-pass-rate)
|
||||||
$(PY) -m bench.batteries.runner --battery 5s --sub synthesis \
|
$(PY) -m bench.batteries.runner --battery 5s --sub synthesis \
|
||||||
--fixtures bench/fixtures/5s/synthesis-v1.jsonl
|
--fixtures bench/fixtures/5s/synthesis-v1.jsonl
|
||||||
|
|
||||||
bench-5s-semiotics: bootstrap
|
bench-5s-semiotics: ## 5S Semiotics (invariance-under-swap)
|
||||||
$(PY) -m bench.batteries.runner --battery 5s --sub semiotics \
|
$(PY) -m bench.batteries.runner --battery 5s --sub semiotics \
|
||||||
--fixtures bench/fixtures/5s/semiotics-v1.jsonl
|
--fixtures bench/fixtures/5s/semiotics-v1.jsonl
|
||||||
```
|
```
|
||||||
|
|
||||||
Existing `bench-5s` aggregate target picks them up via `--all`
|
The existing `bench-5s` aggregate target picks them up via
|
||||||
once the default fixture map in `runner.py` includes them.
|
`--all` once `_DEFAULT_FIXTURES` in `runner.py` includes them.
|
||||||
|
|
||||||
### 4.4 Tests
|
|
||||||
|
|
||||||
- Replace `test_stub_sub_batteries_return_zero` parametrizations
|
|
||||||
for the three sub-batteries with positive coverage:
|
|
||||||
`test_5s_syllogism_runs`, `test_5s_synthesis_runs`,
|
|
||||||
`test_5s_semiotics_runs`.
|
|
||||||
- Add fixture-digest stability test for each new fixture file.
|
|
||||||
|
|
||||||
### 4.5 Bumps
|
|
||||||
|
|
||||||
- New fixture files (additive). Phase 1a fixture digests stay
|
|
||||||
pinned.
|
|
||||||
- Runner-stub replacements in `b_5s.py`.
|
|
||||||
- New Makefile targets.
|
|
||||||
- No schema changes. No `governance_policy_hash` changes.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Out of scope
|
## 6. Tests
|
||||||
|
|
||||||
- 5T Phase 1b (Triangulation, Truthtables, Transitivity, Time —
|
Replace stub-zero parametrizations for the three sub-batteries.
|
||||||
ticket #000024).
|
|
||||||
- 5F battery (Function, Finetuning, Falsification, Formulate,
|
Required positive coverage:
|
||||||
Feedback Loop — ticket #000025).
|
|
||||||
- 5R battery (workspace operators) — depends on SelfModel
|
```
|
||||||
(#000014, landed) + memory_root (#000017, landed); reserved as
|
test_5s_syllogism_runs
|
||||||
Phase 2 of #000021.
|
test_5s_synthesis_runs
|
||||||
|
test_5s_semiotics_runs
|
||||||
|
test_5s_syllogism_fixture_digest_stable
|
||||||
|
test_5s_synthesis_fixture_digest_stable
|
||||||
|
test_5s_semiotics_fixture_digest_stable
|
||||||
|
test_5s_phase1a_digest_unchanged # guard against regression
|
||||||
|
```
|
||||||
|
|
||||||
|
Modality-safety tests (carrier-schema future-readiness):
|
||||||
|
|
||||||
|
```
|
||||||
|
test_5s_fixture_accepts_carrier_metadata
|
||||||
|
test_5s_text_runner_rejects_unsupported_nontext_carrier_cleanly
|
||||||
|
test_5s_semiotics_synonym_swap_preserves_invariant_fields
|
||||||
|
test_5s_semiotics_bad_swap_fails
|
||||||
|
```
|
||||||
|
|
||||||
|
The last group enforces that the schema accepts carrier metadata
|
||||||
|
today even though only text is executed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Out of scope
|
||||||
|
|
||||||
|
- Cross-domain π* composition for 5S (covered by ticket #000015,
|
||||||
|
already landed).
|
||||||
|
- Multimodal 5S execution (depends on #000013 spatial-temporal
|
||||||
|
substrate + π* image / audio / world canonicalizers; future
|
||||||
|
tickets).
|
||||||
|
- 5T Phase 1b (ticket #000024).
|
||||||
|
- 5F battery (ticket #000025).
|
||||||
|
- 5R battery — depends on SelfModel (#000014, landed) +
|
||||||
|
memory_root (#000017, landed); reserved as Phase 2 of #000021.
|
||||||
- Cross-corpus expansion (Phase 3 of #000021).
|
- Cross-corpus expansion (Phase 3 of #000021).
|
||||||
|
- Hidden-channel **generation** or **concealment** tooling.
|
||||||
|
Detection-only is the boundary.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Risks & open questions
|
## 8. Risks & mitigations
|
||||||
|
|
||||||
- **Syllogism fixture quality.** Hand-curated chains can over-fit
|
| Risk | Mitigation |
|
||||||
to arborist's current claim-lattice parser. Mitigation: pull
|
|---|---|
|
||||||
from at least three different test files so coverage spans
|
| Syllogism overfits to toy chains | Pull from ≥3 different test-source families; include invalid converses, missing-premise, unsupported-evidence-pointer failures. |
|
||||||
multiple authoring styles.
|
| Semiotics swaps accidentally change meaning | Use only `concept_relations.relation_kind='synonym'` for v1 positives; include explicit non-equivalent negative swaps. |
|
||||||
- **Synthesis variance.** Depends on retrieval which has
|
| Synthesis retrieval variance | Pin `shard_set` + `retrieval_mode` + `expected_fact_ids` in fixture meta; compare derivation shape, not exact prose. |
|
||||||
per-shard variance. Mitigation: pin shard set in the fixture
|
| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers rejected cleanly in v1; multimodal execution deferred to #000013 / #000015 follow-ups. |
|
||||||
meta; bench-maxing 5pp signal floor accounts for noise.
|
| Steganography → dual-use tooling | v1 does not generate, hide, or recover covert messages. Future fixtures framed as defensive detection / flagging. |
|
||||||
- **Semiotics blast radius.** A bad swap fixture could mask
|
|
||||||
genuine drift if the swap is too aggressive (e.g., swaps that
|
|
||||||
also change semantic meaning, not just surface). Mitigation:
|
|
||||||
curate from `concept_relations` of kind `synonym` only — those
|
|
||||||
are corpus-derived equivalences with known semantic preservation.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Status
|
## 9. Closure criterion
|
||||||
|
|
||||||
|
#000023 closes when:
|
||||||
|
|
||||||
|
1. `run_syllogism`, `run_synthesis`, `run_semiotics` are all non-stub.
|
||||||
|
2. Each ships ≥30 deterministic fixtures with stable `fixture_digest`.
|
||||||
|
3. Fixtures include `carrier` / `domain` / `pi_star_ref` /
|
||||||
|
`loss_report_refs` / `modality_notes` fields.
|
||||||
|
4. Unsupported non-text carriers fail/skip explicitly with
|
||||||
|
`reason="unsupported_carrier"`.
|
||||||
|
5. `make bench-5s` runs all five 5S sub-batteries (Syntax,
|
||||||
|
Semantics, Syllogism, Synthesis, Semiotics).
|
||||||
|
6. Phase 1a fixture digests (`syntax-v1.jsonl`,
|
||||||
|
`semantics-v1.jsonl`) unchanged.
|
||||||
|
7. No schema, governance hash, or canonicalization-version changes.
|
||||||
|
8. Tests cover runner execution, fixture-digest stability, and
|
||||||
|
carrier-schema future-readiness.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Status
|
||||||
|
|
||||||
**Open · awaiting go/no-go.** Three sub-batteries × ~30 fixtures
|
**Open · awaiting go/no-go.** Three sub-batteries × ~30 fixtures
|
||||||
× one runner each ≈ 600-800 lines of code + fixtures. Estimated
|
× one runner each ≈ 800-1000 lines code + fixtures. Mid-sized
|
||||||
mid-sized session.
|
session.
|
||||||
|
|
||||||
Closure criterion: all three runners non-stub, fixture files
|
The architectural correction from the 2026-05-08 review:
|
||||||
landed with version pins, `make bench-5s` runs all five 5S
|
**Semiotics, Syntax, and Semantics are defined over sign-bearing
|
||||||
sub-batteries (Syntax, Semantics, Syllogism, Synthesis, Semiotics)
|
carriers, not just written language.** Phase 1b ships the text
|
||||||
and emits a JSON result per sub-battery. Existing Phase 1a
|
implementation; the fixture schema preserves the future path
|
||||||
fixtures keep their digests.
|
into spatial-temporal world models, multimodal AGI evaluation,
|
||||||
|
and defensive hidden-channel detection.
|
||||||
|
|
|
||||||
|
|
@ -1,160 +1,318 @@
|
||||||
# Ticket #000024 — 5T Phase 1b: complete sub-batteries + align vocabulary with Dav1DPrometheus
|
# Ticket #000024 — 5T Phase 1b: vocabulary alignment + carrier-aware completion
|
||||||
|
|
||||||
**Status:** open · awaiting go/no-go
|
**Status:** open · awaiting go/no-go
|
||||||
**Opened:** 2026-05-07
|
**Opened:** 2026-05-07
|
||||||
**Scope:** Complete the 5T battery — rename existing sub-batteries
|
**Revised:** 2026-05-08 (cross-modality correction folded in per
|
||||||
to match Dav1DPrometheus's authoritative vocabulary
|
review response)
|
||||||
(Transfer → Transfer Learning, Truth → Truthtables, Timing →
|
**Scope:** Two coupled jobs:
|
||||||
Time), and ship real implementations + fixtures for the four
|
1. Align 5T vocabulary with Dav1DPrometheus's authoritative
|
||||||
remaining stub sub-batteries (Triangulation, Truthtables,
|
formulation (Transfer → Transfer Learning, Truth → Truthtables,
|
||||||
Transitivity, Time).
|
Timing → Time).
|
||||||
**Audience:** fox + future bench authors + anyone composing a v8
|
2. Replace the four 5T zero-task stubs (Triangulation, Truthtables,
|
||||||
fitness target across the 5T axes.
|
Transitivity, Time) with carrier-aware Phase-1b runners.
|
||||||
**Hard constraint:** the rename is a NEW fixture-version landing
|
**Audience:** fox + future bench authors + downstream v8 / v7-W /
|
||||||
(`v1` → `v2`); existing `transfer-v1.jsonl` stays in place under
|
multimodal work needing fitness targets across 5T axes.
|
||||||
its sub-battery name `transfer` so #000021 Phase 1a digests don't
|
**Hard constraint:** Phase 1a `transfer-v1.jsonl` digest stays
|
||||||
shift retroactively. New work uses the renamed identifiers.
|
pinned; new work uses `v2` fixture filenames and renamed sub-battery
|
||||||
|
identifiers. No schema bump, no governance hash bump, no
|
||||||
|
canonicalization-version bump. Unsupported non-text carriers MUST
|
||||||
|
fail or skip explicitly. No LLM-as-judge in any 5T runner.
|
||||||
|
|
||||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ts: Transitivity,
|
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ts: Transitivity,
|
||||||
Transfer Learning, Triangulation, Truthtables, Time."*
|
Transfer Learning, Triangulation, Truthtables, Time."*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Problem statement
|
## 1. Corrected purpose
|
||||||
|
|
||||||
Two coupled gaps:
|
**Vocabulary correction is not cosmetic.** Each rename pins a more
|
||||||
|
operationally precise concept:
|
||||||
|
|
||||||
### 1.1 Vocabulary drift from authoritative framework
|
| arborist Phase 1a | Dav1DPrometheus | Why his is right |
|
||||||
|
|
||||||
The SQD whitepaper used: **Transfer · Triangulate · Timing ·
|
|
||||||
Transitivity · Truth.**
|
|
||||||
|
|
||||||
Dav1DPrometheus's published framework uses: **Transitivity ·
|
|
||||||
**Transfer Learning** · Triangulation · **Truthtables** · **Time**.**
|
|
||||||
|
|
||||||
Three differences:
|
|
||||||
|
|
||||||
| arborist (current) | Dav1DPrometheus | Why his is right |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; "Transfer Learning" pins the concrete ML primitive. |
|
| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; pins the concrete ML primitive (cross-task / cross-domain method preservation). |
|
||||||
| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the concrete logical-scenario coverage primitive. |
|
| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the explicit logical-state coverage primitive. |
|
||||||
| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, which is the actual semantic. |
|
| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, lineage, memory, stale-state handling. |
|
||||||
|
|
||||||
arborist landed Phase 1a with the SQD-whitepaper names. Per memory
|
The architectural correction (folded in 2026-05-08):
|
||||||
note `dav1dprometheus_framework.md`, his wording is authoritative —
|
|
||||||
arborist honors it.
|
|
||||||
|
|
||||||
### 1.2 Four stub sub-batteries return zero tasks
|
> **5T must not be permanently language-only or prose-only.
|
||||||
|
> Transitivity, Transfer Learning, Triangulation, Truthtables, and
|
||||||
|
> Time must eventually operate across text, code, tables, images,
|
||||||
|
> audio, sensor streams, spatial state, world-model state, and
|
||||||
|
> hidden or alternate media carriers.**
|
||||||
|
|
||||||
Same as 5S (ticket #000023): Triangulation, Truthtables,
|
Phase 1b implements **text / claim-lattice / memory-root** carriers
|
||||||
Transitivity, Time all currently return zero-task `BatteryResult`
|
only. The fixture schema is carrier-aware so future phases can
|
||||||
objects via `_stub_result` in `bench/batteries/b_5t.py`. Without
|
attach code / image / audio / sensor / world canonicalizers via
|
||||||
real implementations:
|
the π* registry without re-authoring fixtures.
|
||||||
|
|
||||||
| Sub-battery | Failure mode under stubs |
|
|
||||||
|---|---|
|
|
||||||
| **Triangulation** | No bench for "do independent verifiers agree?" — directly the v7 §11.4 cross-source consistency claim. |
|
|
||||||
| **Truthtables** | No coverage of "for these N propositional variables, does the system handle all 2^N cases?" — gates v8 selection's ability to certify logical-completeness claims. |
|
|
||||||
| **Transitivity** | No bench for "if A→B and B→C, does the system derive A→C?" — multi-step entailment is a SelfModel capability claim with no current measurement. |
|
|
||||||
| **Time** | No bench for "does the system maintain temporal context across queries?" — directly relates to memory_root + SelfModel's lineage chain, both shipped but unbenchmarked. |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. Design choices
|
## 2. Carrier-aware 5T definitions
|
||||||
|
|
||||||
### 2.1 Rename strategy
|
| Sub-battery | Definition (carrier-general) | Phase 1b carrier |
|
||||||
|
|---|---|---|
|
||||||
|
| **Transfer Learning** | Whether a learned structure or method transfers across a task, domain, or carrier. Future: text rule → table task; diagram relation → textual derivation; world-state pattern → claim-lattice conclusion. | text / claim_lattice |
|
||||||
|
| **Triangulation** | Independent evidence, verifier, or modality paths converging on the same claim state. Future: text + table + chart agreement; image/OCR + caption + metadata; sensor + event log + textual report. | claim_lattice (existing four verifier strategies) |
|
||||||
|
| **Transitivity** | Relation chains preserving valid inference across steps. Future: spatial + visual relations; code dependency + package metadata; sensor temporal-ordering. | claim_lattice (typed relation graph) |
|
||||||
|
| **Truthtables** | Exhaustive logical-state coverage. Future: visual state toggles; world-state propositions; sensor Boolean predicates; tool precondition matrices. | propositional_logic (N=2..4 variables) |
|
||||||
|
| **Time** | Temporal-context maintenance, lineage, memory, stale-state handling. Future: video frames; sensor streams; world-state transitions; tool/action DAG timelines. | memory_snapshot / selfmodel_snapshot |
|
||||||
|
|
||||||
**A. Hard rename in place.** Update `b_5t.py` function names and
|
Time is the **first sub-battery where v8 substrate components**
|
||||||
fixture filenames; existing Phase 1a digests change. Cleanest
|
(SelfModel #000014 + memory_root #000017) become measurable bench
|
||||||
forward but invalidates Phase 1a results.
|
targets. That is structurally significant: until #000024, those
|
||||||
|
substrates were audit-chain artifacts only; now they have a
|
||||||
**B. Coexist v1 + v2.** Keep `transfer-v1.jsonl` + `run_transfer`
|
fitness signal.
|
||||||
function in place; add `transfer-learning-v2.jsonl` +
|
|
||||||
`run_transfer_learning`. New work uses v2 names. Old `bench-5t`
|
|
||||||
target still runs against v1.
|
|
||||||
|
|
||||||
**C. Soft rename (alias).** Keep file names; add alias mapping in
|
|
||||||
the runner so `--sub transfer-learning` resolves to `run_transfer`.
|
|
||||||
Code stays ugly forever.
|
|
||||||
|
|
||||||
**Recommendation:** B. Phase 1a was honest at the time it landed
|
|
||||||
(SQD whitepaper was the only source then); we don't retroactively
|
|
||||||
break it. New work explicitly uses Dav1DPrometheus's vocabulary.
|
|
||||||
|
|
||||||
### 2.2 Fixture sourcing
|
|
||||||
|
|
||||||
- **Triangulation** — replay arborist's existing four-strategy
|
|
||||||
verifier (quote/span/entity/paraphrase) on a fixture set;
|
|
||||||
metric is the per-task agreement rate. Already-shipped verifier
|
|
||||||
surface; fixtures are easy.
|
|
||||||
- **Truthtables** — small-N propositional fixtures (N=2..4); each
|
|
||||||
fixture lists all 2^N input combinations + the expected output;
|
|
||||||
arborist's claim-lattice has to evaluate every row. New surface
|
|
||||||
(currently arborist doesn't do propositional eval); requires a
|
|
||||||
thin wrapper.
|
|
||||||
- **Transitivity** — multi-step claim chains pulled from existing
|
|
||||||
claim-lattice tests; metric is full-chain-pass-rate.
|
|
||||||
- **Time** — multi-query sequences against the same shard set;
|
|
||||||
metric is "does memory_root snapshot N+1 preserve information
|
|
||||||
from snapshot N?" Direct integration with memory-root (#000017,
|
|
||||||
landed).
|
|
||||||
|
|
||||||
### 2.3 Cross-reference to memory-root and SelfModel
|
|
||||||
|
|
||||||
The Time sub-battery implementation reads memory_records and
|
|
||||||
selfmodel_records to assert temporal-context preservation. This
|
|
||||||
makes #000024 the first ticket where the v8 substrate
|
|
||||||
(SelfModel + memory_root) gets a measurable bench target, not
|
|
||||||
just an audit-chain artifact.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Recommendation
|
## 3. Fixture schema (carrier-aware)
|
||||||
|
|
||||||
Two-phase landing:
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-<sub>-NNN",
|
||||||
|
"battery": "5t",
|
||||||
|
"sub_battery": "time",
|
||||||
|
"version": "v1",
|
||||||
|
|
||||||
**Phase 1b.1 — Rename + cosmetic alignment.** Add v2 fixture file
|
"carrier": "memory_snapshot",
|
||||||
names + new sub-battery aliases. Keep v1 in place. ~100 lines.
|
"domain": "memory_root",
|
||||||
|
"pi_star_ref": "pi_memory_v1",
|
||||||
|
"loss_report_refs": [],
|
||||||
|
"modality_notes": "Phase 1b: text/claim-lattice/memory-root only; future carriers may include code/table/image/audio/sensor/world.",
|
||||||
|
|
||||||
**Phase 1b.2 — Implement four sub-batteries.** Triangulation
|
"task": { ... }
|
||||||
first (existing surface), then Transitivity (lexical), then
|
}
|
||||||
Truthtables (new wrapper), then Time (memory-root-integrated).
|
|
||||||
~600-800 lines.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Implementation sketch
|
|
||||||
|
|
||||||
### 4.1 Fixture files (new)
|
|
||||||
|
|
||||||
```
|
|
||||||
bench/fixtures/5t/transfer-learning-v2.jsonl # rename of transfer-v1
|
|
||||||
bench/fixtures/5t/triangulation-v1.jsonl
|
|
||||||
bench/fixtures/5t/truthtables-v1.jsonl
|
|
||||||
bench/fixtures/5t/transitivity-v1.jsonl
|
|
||||||
bench/fixtures/5t/time-v1.jsonl
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`transfer-v1.jsonl` stays in place; `bench-5t-transfer` still runs
|
Phase 1b allowed carriers: `text`, `claim_lattice`,
|
||||||
it. Phase 1a tests stay green.
|
`memory_snapshot`, `selfmodel_snapshot`, `verifier_strategies`,
|
||||||
|
`propositional_logic`, `relation_graph`. Other values →
|
||||||
|
`reason="unsupported_carrier"`.
|
||||||
|
|
||||||
### 4.2 Runner additions
|
---
|
||||||
|
|
||||||
In `bench/batteries/b_5t.py`, real implementations replace stubs:
|
## 4. Per-sub-battery design
|
||||||
|
|
||||||
- `run_transfer_learning(fixtures_path)` — same logic as
|
### 4.1 Transfer Learning (rename of Phase 1a `transfer`)
|
||||||
`run_transfer`; alias.
|
|
||||||
- `run_triangulation(fixtures_path)` — for each task, call
|
|
||||||
arborist's four verifier strategies on (claim, evidence) pairs
|
|
||||||
and report agreement rate.
|
|
||||||
- `run_truthtables(fixtures_path)` — for each task, evaluate the
|
|
||||||
claim-lattice over 2^N input combinations.
|
|
||||||
- `run_transitivity(fixtures_path)` — chase A→B, B→C through the
|
|
||||||
claim-lattice; assert C derives.
|
|
||||||
- `run_time(fixtures_path)` — multi-query memory_root preservation;
|
|
||||||
reads `memory_records` to assert continuity.
|
|
||||||
|
|
||||||
### 4.3 Makefile additions
|
**Metric:** `transfer_learning_success_rate`.
|
||||||
|
|
||||||
|
**Backward compat:** Keep `transfer-v1.jsonl`, `run_transfer`,
|
||||||
|
`bench-5t-transfer` in place — Phase 1a digest stays pinned. Add
|
||||||
|
`transfer-learning-v2.jsonl`, `run_transfer_learning`,
|
||||||
|
`bench-5t-transfer-learning` for new work.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-transfer-learning-001",
|
||||||
|
"version": "v2",
|
||||||
|
"carrier": "text",
|
||||||
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"source_task": {
|
||||||
|
"pattern": "A supports B when cited span contains anchor B.",
|
||||||
|
"example": "Release-date claim with date anchor."
|
||||||
|
},
|
||||||
|
"target_task": {
|
||||||
|
"pattern": "Founder-date claim with date anchor.",
|
||||||
|
"expected_transfer": true
|
||||||
|
},
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Triangulation
|
||||||
|
|
||||||
|
**Metric:** `triangulation_agreement_rate`,
|
||||||
|
`strategy_disagreement_rate`.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-triangulation-001",
|
||||||
|
"carrier": "claim_lattice",
|
||||||
|
"domain": "verifier_strategies",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"claim": "Back to the Future was released on July 3, 1985.",
|
||||||
|
"evidence": ["E1", "E2"],
|
||||||
|
"strategies": ["quote", "span", "entity", "paraphrase"],
|
||||||
|
"expected_agreement_min": 0.75
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Runner:** invokes arborist's existing four verifier strategies
|
||||||
|
on (claim, evidence) pairs, computes pairwise agreement, passes
|
||||||
|
when agreement ≥ `expected_agreement_min`.
|
||||||
|
|
||||||
|
**Future-modality hook:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"future_modality_paths": [
|
||||||
|
{"carrier": "table", "role": "structured evidence"},
|
||||||
|
{"carrier": "image", "role": "visual confirmation"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
v1 runner ignores; fixture is forward-compatible.
|
||||||
|
|
||||||
|
### 4.3 Transitivity
|
||||||
|
|
||||||
|
**Metric:** `full_chain_pass_rate`.
|
||||||
|
|
||||||
|
**Critical correction:** not all relations are transitive. Phase
|
||||||
|
1b ships a typed-relation-whitelist:
|
||||||
|
|
||||||
|
```
|
||||||
|
implies
|
||||||
|
subset_of
|
||||||
|
ancestor_of
|
||||||
|
before
|
||||||
|
less_than
|
||||||
|
```
|
||||||
|
|
||||||
|
Runners reject mixed-relation chains and non-transitive relations
|
||||||
|
(`related_to`, `causes` without domain constraints, etc.).
|
||||||
|
|
||||||
|
**Positive fixture:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-transitivity-001",
|
||||||
|
"carrier": "claim_lattice",
|
||||||
|
"domain": "relation_graph",
|
||||||
|
"pi_star_ref": "pi_relation_graph_v1",
|
||||||
|
"edges": [
|
||||||
|
{"from": "A", "to": "B", "relation": "implies"},
|
||||||
|
{"from": "B", "to": "C", "relation": "implies"}
|
||||||
|
],
|
||||||
|
"query": {"from": "A", "to": "C", "relation": "implies"},
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Negative fixture:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-transitivity-NEG-001",
|
||||||
|
"edges": [
|
||||||
|
{"from": "A", "to": "B", "relation": "related_to"},
|
||||||
|
{"from": "B", "to": "C", "relation": "causes"}
|
||||||
|
],
|
||||||
|
"query": {"from": "A", "to": "C", "relation": "causes"},
|
||||||
|
"expected": "fail",
|
||||||
|
"reason": "mixed relation types do not license transitive closure"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 Truthtables
|
||||||
|
|
||||||
|
**Metric:** `truth_table_coverage_rate`, `row_accuracy_rate`.
|
||||||
|
|
||||||
|
**Constraint:** N=2..4 variables only (avoids combinatorial
|
||||||
|
blowup; keeps logical coverage real).
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-truthtables-001",
|
||||||
|
"carrier": "claim_lattice",
|
||||||
|
"domain": "propositional_logic",
|
||||||
|
"pi_star_ref": "pi_truth_table_v1",
|
||||||
|
"variables": ["A", "B"],
|
||||||
|
"expression": "A AND B",
|
||||||
|
"rows": [
|
||||||
|
{"inputs": {"A": false, "B": false}, "expected": false},
|
||||||
|
{"inputs": {"A": false, "B": true}, "expected": false},
|
||||||
|
{"inputs": {"A": true, "B": false}, "expected": false},
|
||||||
|
{"inputs": {"A": true, "B": true}, "expected": true}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Runner:** parse expression → enumerate rows → evaluate each
|
||||||
|
deterministically → fixture fails if any row mismatches.
|
||||||
|
Deterministic parser; no LLM.
|
||||||
|
|
||||||
|
### 4.5 Time
|
||||||
|
|
||||||
|
**Metric:** `temporal_context_preservation_rate`,
|
||||||
|
`stale_marking_rate`, `memory_lineage_pass_rate`.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5t-time-001",
|
||||||
|
"carrier": "memory_snapshot",
|
||||||
|
"domain": "memory_root",
|
||||||
|
"pi_star_ref": "pi_memory_v1",
|
||||||
|
"snapshots": [
|
||||||
|
{"snapshot_id": "S0", "memory_root": "M0", "facts": ["X is true as of t0"]},
|
||||||
|
{"snapshot_id": "S1", "memory_root": "M1", "facts": ["X is revised as of t1"]}
|
||||||
|
],
|
||||||
|
"expected": {
|
||||||
|
"preserves_prior": true,
|
||||||
|
"marks_stale": ["X is true as of t0"],
|
||||||
|
"current_root": "M1"
|
||||||
|
},
|
||||||
|
"policy_assumption": {
|
||||||
|
"memory_root_binding": "advisory",
|
||||||
|
"selfmodel_integration": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Runner:** reads `memory_records` (and `selfmodel_records` where
|
||||||
|
needed) → compares snapshot N and N+1 → asserts temporal
|
||||||
|
preservation, stale marking, lineage continuity.
|
||||||
|
|
||||||
|
**Future-time-carriers (documented, not executed in v1):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"future_time_carriers": [
|
||||||
|
{"carrier": "video", "domain": "frame_sequence"},
|
||||||
|
{"carrier": "sensor", "domain": "time_series"},
|
||||||
|
{"carrier": "world", "domain": "state_transition_graph"}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Required file changes
|
||||||
|
|
||||||
|
### 5.1 Fixture files
|
||||||
|
|
||||||
|
```
|
||||||
|
bench/fixtures/5t/transfer-learning-v2.jsonl # new ≥30 tasks
|
||||||
|
bench/fixtures/5t/triangulation-v1.jsonl # new ≥30 tasks
|
||||||
|
bench/fixtures/5t/truthtables-v1.jsonl # new ≥30 tasks (N=2..4)
|
||||||
|
bench/fixtures/5t/transitivity-v1.jsonl # new ≥30 tasks (whitelist)
|
||||||
|
bench/fixtures/5t/time-v1.jsonl # new ≥30 tasks (memory)
|
||||||
|
|
||||||
|
bench/fixtures/5t/transfer-v1.jsonl # UNCHANGED (Phase 1a digest pinned)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 Runner functions
|
||||||
|
|
||||||
|
In `bench/batteries/b_5t.py`:
|
||||||
|
|
||||||
|
- Keep `run_transfer` (Phase 1a) intact.
|
||||||
|
- Add `run_transfer_learning`, `run_triangulation`,
|
||||||
|
`run_truthtables`, `run_transitivity`, `run_time` as non-stub
|
||||||
|
implementations.
|
||||||
|
|
||||||
|
### 5.3 Make targets
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
bench-5t-transfer-learning: bootstrap
|
bench-5t-transfer-learning: bootstrap
|
||||||
|
|
@ -178,61 +336,100 @@ bench-5t-time: bootstrap
|
||||||
--fixtures bench/fixtures/5t/time-v1.jsonl
|
--fixtures bench/fixtures/5t/time-v1.jsonl
|
||||||
```
|
```
|
||||||
|
|
||||||
Existing `bench-5t` aggregate keeps `transfer` (v1) for back-compat.
|
`bench-5t` aggregate keeps `transfer` (v1) for backward
|
||||||
|
compatibility. New canonical aggregate is `bench-5t-v2` covering
|
||||||
### 4.4 Tests
|
the five Dav1DPrometheus sub-batteries; or operators wire
|
||||||
|
`bench-5t` to the v2 names once the rename has stabilized.
|
||||||
- Add `test_5t_transfer_learning_alias` — confirms new name resolves
|
|
||||||
to same surface as `transfer`.
|
|
||||||
- Add positive tests for each new sub-battery.
|
|
||||||
- Update stub-test parametrization to remove the four sub-batteries
|
|
||||||
that get real implementations.
|
|
||||||
|
|
||||||
### 4.5 Bumps
|
|
||||||
|
|
||||||
- New fixture files (additive).
|
|
||||||
- New runner functions; old ones (`run_transfer`,
|
|
||||||
`run_triangulate`, etc.) stay in place for back-compat.
|
|
||||||
- New Makefile targets.
|
|
||||||
- No schema changes.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Out of scope
|
## 6. Tests
|
||||||
|
|
||||||
- 5S Phase 1b — ticket #000023.
|
```
|
||||||
- 5F battery (entirely new) — ticket #000025.
|
test_5t_transfer_learning_runs
|
||||||
- 5R battery (workspace operators) — Phase 2 of #000021.
|
test_5t_triangulation_runs
|
||||||
- Retiring the v1 names. Once v2 is the standard, deprecate v1
|
test_5t_truthtables_runs
|
||||||
in a follow-up ticket; this ticket leaves both live.
|
test_5t_transitivity_runs
|
||||||
|
test_5t_time_runs
|
||||||
|
test_5t_v1_transfer_digest_unchanged # guard regression
|
||||||
|
test_5t_fixture_accepts_carrier_metadata
|
||||||
|
test_5t_text_runner_rejects_unsupported_nontext_carrier_cleanly
|
||||||
|
test_5t_time_reads_memory_root_fixture
|
||||||
|
test_5t_transitivity_rejects_non_whitelisted_relation
|
||||||
|
test_5t_truthtables_N_le_4_only
|
||||||
|
```
|
||||||
|
|
||||||
|
Stable-digest tests for each new fixture file. Remove the four
|
||||||
|
implemented sub-batteries from any "stub returns zero"
|
||||||
|
parametrization.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Risks & open questions
|
## 7. Out of scope
|
||||||
|
|
||||||
- **Truthtables surface.** arborist doesn't currently do
|
- Multimodal 5T execution (depends on #000013 spatial-temporal
|
||||||
propositional evaluation; building the wrapper is real work.
|
substrate + π* image / audio / world canonicalizers).
|
||||||
Defer if scope compresses; ship the other three first.
|
- 5S Phase 1b (ticket #000023).
|
||||||
- **Time integration depth.** memory_root snapshots are advisory
|
- 5F battery (ticket #000025).
|
||||||
(default policy off). The Time bench can still measure
|
- 5R battery — Phase 2 of #000021.
|
||||||
preservation across snapshots even when binding is off, but
|
- Retiring v1 names. Once v2 is the standard, deprecate v1 in a
|
||||||
results may not reflect production behavior under different
|
follow-up; this ticket leaves both live.
|
||||||
policies. Document the policy assumption in the fixture meta.
|
- Hidden-channel **generation**. Defensive detection only.
|
||||||
- **Backward compatibility.** Old `bench-5t` (v1 names) stays
|
|
||||||
alongside new `bench-5t-*` (v2 names). Risk of confusion;
|
|
||||||
mitigate by clear naming + a `docs/bench-vocabulary.md` note
|
|
||||||
if needed.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Status
|
## 8. Risks & mitigations
|
||||||
|
|
||||||
**Open · awaiting go/no-go.** Mid-to-large surface; biggest sub-
|
| Risk | Mitigation |
|
||||||
battery (Time) needs memory-root integration. Recommended
|
|---|---|
|
||||||
implementation sequence: rename first, then sub-batteries in the
|
| Vocabulary confusion (v1 vs v2) | Keep v1 transfer fixtures; add v2 transfer-learning fixtures; document Dav1DPrometheus canonical vocabulary. |
|
||||||
order Triangulation → Transitivity → Time → Truthtables.
|
| Truthtables scope blowup | N=2..4 only; deterministic parser; no LLM-as-judge. |
|
||||||
|
| Transitivity overclaims | Typed relation graph; transitive-relation whitelist; negative fixtures for non-transitive relation types. |
|
||||||
|
| Time policy ambiguity | Fixture meta includes `memory_root_binding` policy assumption; runner reports policy mismatch. |
|
||||||
|
| Premature multimodal scope creep | Carrier metadata now; non-text carrier support later; unsupported carriers fail/skip explicitly. |
|
||||||
|
|
||||||
Closure criterion: all five 5T sub-batteries (Transfer Learning,
|
---
|
||||||
Triangulation, Truthtables, Transitivity, Time) have non-stub
|
|
||||||
runners + fixture files; `make bench-5t` runs the full set under
|
## 9. Closure criterion
|
||||||
v2 names; v1 `bench-5t-transfer` still passes.
|
|
||||||
|
#000024 closes when:
|
||||||
|
|
||||||
|
1. Dav1DPrometheus vocabulary present: Transfer Learning,
|
||||||
|
Triangulation, Truthtables, Transitivity, Time.
|
||||||
|
2. Legacy `transfer-v1.jsonl` digest unchanged.
|
||||||
|
3. New `transfer-learning-v2.jsonl` lands.
|
||||||
|
4. Triangulation runner non-stub.
|
||||||
|
5. Transitivity runner non-stub with typed-relation whitelist.
|
||||||
|
6. Truthtables runner non-stub with N=2..4 cap.
|
||||||
|
7. Time runner non-stub and reads `memory_records` /
|
||||||
|
`selfmodel_records` surfaces.
|
||||||
|
8. All new fixtures include `carrier` / `domain` / `pi_star_ref` /
|
||||||
|
`loss_report_refs` / `modality_notes`.
|
||||||
|
9. Unsupported future carriers fail or skip explicitly.
|
||||||
|
10. Make targets for each sub-battery pass.
|
||||||
|
11. Phase 1a fixture digests unchanged.
|
||||||
|
12. No schema, governance hash, or canonicalization-version changes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Status
|
||||||
|
|
||||||
|
**Open · awaiting go/no-go.** Two-phase landing recommended:
|
||||||
|
|
||||||
|
- **Phase 1b.1 — vocabulary alignment.** Add v2 fixture file names
|
||||||
|
+ new sub-battery aliases. Keep v1 in place. ~150 lines.
|
||||||
|
- **Phase 1b.2 — implement four sub-batteries.** Triangulation
|
||||||
|
first (existing surface), Transitivity second (typed-relation
|
||||||
|
whitelist), Time third (memory-root integrated), Truthtables
|
||||||
|
last (new propositional wrapper). ~700-900 lines + fixtures.
|
||||||
|
|
||||||
|
The architectural correction:
|
||||||
|
|
||||||
|
> **5T evaluates reasoning transformations across states, domains,
|
||||||
|
> evidence paths, logical cases, and time. Those transformations
|
||||||
|
> can live in text today, but must be representable across any
|
||||||
|
> sign-bearing or state-bearing carrier later.**
|
||||||
|
|
||||||
|
This makes #000024 the first ticket where v8 substrate becomes
|
||||||
|
measurable bench territory and the path to spatial-temporal /
|
||||||
|
multimodal / world-model AGI evaluation stays open.
|
||||||
|
|
|
||||||
|
|
@ -1,190 +1,376 @@
|
||||||
# Ticket #000025 — 5F battery: Function, Finetuning, Falsification, Formulate, Feedback Loop
|
# Ticket #000025 — 5F battery: Function · Finetuning · Falsification · Formulate · Feedback Loop
|
||||||
|
|
||||||
**Status:** open · awaiting go/no-go
|
**Status:** open · awaiting go/no-go
|
||||||
**Opened:** 2026-05-07
|
**Opened:** 2026-05-07
|
||||||
**Scope:** Open the 5F battery — Dav1DPrometheus's third evaluation
|
**Revised:** 2026-05-08 (cross-modality + state-space synthesis
|
||||||
axis, missing entirely from arborist before this ticket. Each sub-
|
folded in per review response)
|
||||||
battery measures a distinct dimension of agent quality:
|
**Scope:** Open the 5F battery — Dav1DPrometheus's
|
||||||
**Function** (task alignment), **Finetuning** (adaptation
|
operational-quality axis, missing entirely from arborist before
|
||||||
efficiency), **Falsification** (error detection + correction),
|
this ticket. Five sub-batteries: **Function** (task alignment),
|
||||||
**Formulate** (logical structuring), **Feedback Loop** (continuous
|
**Finetuning** (adaptation efficiency), **Falsification** (error
|
||||||
improvement).
|
detection + correction), **Formulate** (logical structuring),
|
||||||
**Audience:** fox + future bench authors. Once landed, 5F + 5S + 5T
|
**Feedback Loop** (continuous improvement). 5F integrates with
|
||||||
together form the complete non-embodied AGI evaluation surface
|
existing arborist surfaces — `providence_cache.falsification_state`,
|
||||||
|
`audit_events`, `memory_records` (#000017, landed),
|
||||||
|
`selfmodel_records` (#000014, landed). No new substrate work.
|
||||||
|
**Audience:** fox + future bench authors. Once 5F lands, 5S + 5T +
|
||||||
|
5F together form the complete non-embodied AGI evaluation surface
|
||||||
Dav1DPrometheus published.
|
Dav1DPrometheus published.
|
||||||
**Hard constraint:** 5F integrates with surfaces that already exist
|
**Hard constraint:** 5F is text-first in Phase 1a but the fixture
|
||||||
in arborist — `providence_cache.falsification_state`, `audit_events`,
|
schema is **carrier-aware**. Unsupported non-text carriers MUST
|
||||||
`memory_records` (#000017), `selfmodel_records` (#000014). Don't
|
fail or skip explicitly. No LLM-as-judge in any 5F runner. No
|
||||||
require new substrate work to land 5F.
|
schema bump, no governance hash bump. Hidden-channel work stays
|
||||||
|
defensive (detection only, never generation).
|
||||||
|
|
||||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Fs: Function,
|
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Fs: Function,
|
||||||
Finetuning, Falsification, Formulate, Feedback Loop."* Honoring his
|
Finetuning, Falsification, Formulate, Feedback Loop."*
|
||||||
legacy.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Problem statement
|
## 1. Why 5F is the operational bridge
|
||||||
|
|
||||||
Dav1DPrometheus's framework names three orthogonal evaluation axes:
|
5S evaluates linguistic / sign-structure precision.
|
||||||
|
5T evaluates temporal / cross-reasoning / logical coverage.
|
||||||
```text
|
**5F evaluates whether the organism acts, adapts, falsifies,
|
||||||
5Ss = linguistic precision (syntax + semantics + syllogism + synthesis + semiotics)
|
formulates, and integrates feedback** — the dimensions that turn a
|
||||||
5Ts = temporal/cross reasoning (transitivity + transfer learning + triangulation + truthtables + time)
|
state-space into an *adaptive* state-space.
|
||||||
5Fs = operational quality (function + finetuning + falsification + formulate + feedback loop)
|
|
||||||
```
|
|
||||||
|
|
||||||
arborist's bench harness (ticket #000021) implements 5S + 5T as
|
|
||||||
fixture-driven batteries. **5F has no representation in arborist.**
|
|
||||||
The original SQD whitepaper omitted it; Dav1DPrometheus's framework
|
|
||||||
is the only source for it.
|
|
||||||
|
|
||||||
Without 5F:
|
Without 5F:
|
||||||
|
|
||||||
| Sub-battery | Why arborist needs it |
|
|
||||||
|---|---|
|
|
||||||
| **Function** | Measures whether the system actually executes its designated task. arborist's QA layer has answer-mode bench coverage but not a clean "did the system DO what it was asked" metric. |
|
|
||||||
| **Finetuning** | Measures speed/efficiency of adaptation. SelfModel parent→child transitions are unbenchmarked. |
|
|
||||||
| **Falsification** | Error-detection accuracy is core to v9.8 admissibility, but `providence_cache.falsification_state` transitions are not benched. |
|
|
||||||
| **Formulate** | Coherence of organized knowledge structures — directly the claim-lattice's job, currently only measured by per-claim verification rate, not chain-level coherence. |
|
|
||||||
| **Feedback Loop** | Integration efficiency for incoming corrections. Memory-root branch projections (#000017) update on snapshot but speed of integration is unmeasured. |
|
|
||||||
|
|
||||||
### 1.1 Why this is a separate ticket from #000021
|
|
||||||
|
|
||||||
#000021 was scoped to the SQD whitepaper's 5S + 5T. Adding 5F
|
|
||||||
would have changed scope mid-design. Treating 5F as its own
|
|
||||||
ticket keeps each battery's story coherent and lets fox approve
|
|
||||||
or punt 5F independently.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Sub-battery semantics (per Dav1DPrometheus)
|
|
||||||
|
|
||||||
### 2.1 Function
|
|
||||||
|
|
||||||
> *Task Alignment: Precision in executing designated tasks.
|
|
||||||
> Operational Effectiveness: Empirical success rate in task execution.*
|
|
||||||
|
|
||||||
Each task: a directive (e.g., "summarize document X"), a target
|
|
||||||
output shape (e.g., claim-lattice with N claims), and an
|
|
||||||
evaluation predicate (does the produced output meet the shape?).
|
|
||||||
Metric: function-pass-rate.
|
|
||||||
|
|
||||||
Maps to arborist surfaces: `arborist ask` invocations under
|
|
||||||
specific `answer_mode` settings.
|
|
||||||
|
|
||||||
### 2.2 Finetuning
|
|
||||||
|
|
||||||
> *Performance Improvement: Quantitative percentage increase in
|
|
||||||
> performance metrics post-finetuning.
|
|
||||||
> Adaptability: Efficiency in adapting to new tasks, measured by
|
|
||||||
> time and resources required.*
|
|
||||||
|
|
||||||
Each task: a parent SelfModel + a target capability claim that
|
|
||||||
the parent fails + a child SelfModel that should pass. Metric:
|
|
||||||
fraction of parent→child transitions where the child's measured
|
|
||||||
metric improves over parent's.
|
|
||||||
|
|
||||||
Maps to arborist surfaces: `selfmodel_records` parent→child chain
|
|
||||||
+ `selfmodel_capability_claims` measured_value tracking.
|
|
||||||
|
|
||||||
### 2.3 Falsification
|
|
||||||
|
|
||||||
> *Error Detection Accuracy: Rate of correctly identifying system
|
|
||||||
> errors. Correction Effectiveness: Success rate in rectifying
|
|
||||||
> identified errors.*
|
|
||||||
|
|
||||||
Each task: a known-bad providence record (planted) + the expected
|
|
||||||
falsification reason. Metric: fraction of planted records the
|
|
||||||
verifier correctly flags + fraction of repair operations that
|
|
||||||
restore correctness.
|
|
||||||
|
|
||||||
Maps to arborist surfaces: `falsification_state` transitions,
|
|
||||||
`providence_repair` audit events, `arborist.qa.repair`.
|
|
||||||
|
|
||||||
### 2.4 Formulate
|
|
||||||
|
|
||||||
> *Logical Structuring: Degree of coherence in organized
|
|
||||||
> knowledge structures.
|
|
||||||
> Framework Clarity: Ease of comprehension and interpretability of
|
|
||||||
> structured knowledge.*
|
|
||||||
|
|
||||||
Each task: a free-text input + the expected claim-lattice
|
|
||||||
structure (N claims, ordered, with pointer IDs). Metric: structural-
|
|
||||||
match-rate (sorted-claim-text + pointer-ID-set agreement, NOT
|
|
||||||
exact-string).
|
|
||||||
|
|
||||||
Maps to arborist surfaces: `arborist.qa.parse_claims` +
|
|
||||||
`arborist.pi_star.claim_lattice`.
|
|
||||||
|
|
||||||
### 2.5 Feedback Loop
|
|
||||||
|
|
||||||
> *Integration Efficiency: Speed and efficacy in assimilating
|
|
||||||
> feedback into system processes.
|
|
||||||
> Continuous Improvement: Measurable performance enhancements
|
|
||||||
> through iterative feedback mechanisms.*
|
|
||||||
|
|
||||||
Each task: a sequence of (operation, observation) pairs forming a
|
|
||||||
feedback chain; the system should integrate observations into
|
|
||||||
subsequent operations. Metric: integration-coverage-rate (fraction
|
|
||||||
of observations that affect a downstream snapshot).
|
|
||||||
|
|
||||||
Maps to arborist surfaces: `memory_branch_summaries` updates +
|
|
||||||
`audit_events` chain delta between snapshots.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Design choices
|
|
||||||
|
|
||||||
### 3.1 Battery vs sibling layer
|
|
||||||
|
|
||||||
**A. Treat 5F as a peer to 5S/5T inside `bench/batteries/`.** Same
|
|
||||||
runner skeleton; new file `b_5f.py`.
|
|
||||||
|
|
||||||
**B. Treat 5F as a separate top-level harness.** New module
|
|
||||||
`bench/quality/`. Distinct because Function/Finetuning are
|
|
||||||
operational rather than substrate-evaluation properties.
|
|
||||||
|
|
||||||
**Recommendation:** A. Keeps the harness uniform; the runner CLI
|
|
||||||
already supports arbitrary battery names. The semantic distinction
|
|
||||||
between 5S/5T (substrate) and 5F (operational) is documented in
|
|
||||||
the ticket but doesn't need a code-level distinction.
|
|
||||||
|
|
||||||
### 3.2 Fixture sourcing per sub-battery
|
|
||||||
|
|
||||||
| Sub-battery | Fixture source |
|
|
||||||
|---|---|
|
|
||||||
| Function | Hand-curated `(directive, expected-shape, evaluator)` triples. |
|
|
||||||
| Finetuning | Generated from `selfmodel_records` chains in actual shards (or test fixtures). |
|
|
||||||
| Falsification | Planted-error fixtures: known-bad providence records with stated falsification reason. |
|
|
||||||
| Formulate | Hand-curated `(text, expected-claim-lattice)` pairs from existing claim-lattice tests. |
|
|
||||||
| Feedback Loop | Multi-snapshot fixture chains; each chain tagged with expected memory-root delta semantics. |
|
|
||||||
|
|
||||||
### 3.3 Phase staging
|
|
||||||
|
|
||||||
Like 5S/5T, 5F lands in two phases:
|
|
||||||
|
|
||||||
- **Phase 1a (this ticket scope):** runner skeleton in `b_5f.py`,
|
|
||||||
~10 seed fixtures per sub-battery, all sub-batteries return real
|
|
||||||
results (no stubs). Get the surface up.
|
|
||||||
- **Phase 1b (follow-up):** expand fixtures to 30-50 per sub-battery
|
|
||||||
for bench-floor signal.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Implementation sketch
|
|
||||||
|
|
||||||
### 4.1 Module + fixtures
|
|
||||||
|
|
||||||
```
|
```
|
||||||
bench/batteries/b_5f.py # new; Function/Finetuning/Falsification/Formulate/Feedback runners
|
state-space exists but may not improve.
|
||||||
bench/fixtures/5f/function-v1.jsonl
|
|
||||||
bench/fixtures/5f/finetuning-v1.jsonl
|
|
||||||
bench/fixtures/5f/falsification-v1.jsonl
|
|
||||||
bench/fixtures/5f/formulate-v1.jsonl
|
|
||||||
bench/fixtures/5f/feedback-loop-v1.jsonl
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4.2 Runner registration
|
With 5F:
|
||||||
|
|
||||||
|
```
|
||||||
|
state-space becomes an adaptive learning organism.
|
||||||
|
```
|
||||||
|
|
||||||
|
That is the operational gap 5F closes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. State-space / time synthesis
|
||||||
|
|
||||||
|
The folded-in correction (2026-05-08 review):
|
||||||
|
|
||||||
|
> **SQD + Merkle-AGI v7 + 5S/5T/5F + arborist together instantiate
|
||||||
|
> an explicit discrete state-space/time in which counting,
|
||||||
|
> mathematics, and logic become auditable operations over committed
|
||||||
|
> state — not just text generated by a latent model.**
|
||||||
|
|
||||||
|
Define an arborist organism state at time `t`:
|
||||||
|
|
||||||
|
```
|
||||||
|
Ω_t = (
|
||||||
|
W_t, workspace state
|
||||||
|
I_t, invariant objects under π*
|
||||||
|
C_t, concept / semiotic state
|
||||||
|
L_t, claim lattice
|
||||||
|
MRoot_t, memory root
|
||||||
|
SMRoot_t, SelfModel root
|
||||||
|
PRoot_t, policy root
|
||||||
|
BRoot_t, benchmark root
|
||||||
|
ARoot_t, audit root
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
A transition is `Ω_{t+1} = T(Ω_t, Δ_t)` where `Δ_t ∈ {new evidence,
|
||||||
|
new claim, new falsification, new benchmark result, new SelfModel
|
||||||
|
update, new memory snapshot, new feedback observation, new repair
|
||||||
|
operation}`. This is a discrete state-space/time — not metaphorical.
|
||||||
|
The system has states, transitions, invariants, events, memory,
|
||||||
|
verification, branching, rollback, selection pressure.
|
||||||
|
|
||||||
|
5F maps onto the transitions:
|
||||||
|
|
||||||
|
| Sub-battery | Transition question |
|
||||||
|
|---|---|
|
||||||
|
| Function | Does the transition do the intended operation? |
|
||||||
|
| Finetuning | Does the organism improve across parent→child states? |
|
||||||
|
| Falsification | Does the organism reject invalid states? |
|
||||||
|
| Formulate | Does the organism convert raw input into structured state? |
|
||||||
|
| Feedback Loop | Does observation at `t` affect state at `t+1`? |
|
||||||
|
|
||||||
|
**Counting** emerges over finite committed sets (claims, verifier
|
||||||
|
passes, memory branches, feedback observations integrated, SelfModel
|
||||||
|
capability upgrades). **Mathematics** emerges over π*-canonical
|
||||||
|
invariants. **Logic** emerges from verifier predicates, truthtables,
|
||||||
|
syllogisms, transitivity, falsification states, admissibility
|
||||||
|
labels. **Time** emerges from ordered state transitions across
|
||||||
|
MemoryRoot / SelfModel / BenchmarkRoot.
|
||||||
|
|
||||||
|
The break from traditional LLM stacks:
|
||||||
|
|
||||||
|
> **Traditional LLMs generate math/logic as behavior. This stack
|
||||||
|
> externalizes math/logic as committed, verifier-addressable state.**
|
||||||
|
|
||||||
|
That is the architectural justification for 5F's existence as a
|
||||||
|
distinct axis.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Carrier-aware 5F definitions
|
||||||
|
|
||||||
|
Common rule:
|
||||||
|
|
||||||
|
> A 5F task evaluates whether the organism can operate correctly
|
||||||
|
> over a carrier, adapt over that carrier, falsify errors in that
|
||||||
|
> carrier, formulate structure from it, or integrate feedback about
|
||||||
|
> it.
|
||||||
|
|
||||||
|
Phase 1a allowed carriers:
|
||||||
|
|
||||||
|
```
|
||||||
|
text · claim_lattice · providence_record ·
|
||||||
|
selfmodel_snapshot · memory_snapshot · audit_event
|
||||||
|
```
|
||||||
|
|
||||||
|
Future carriers:
|
||||||
|
|
||||||
|
```
|
||||||
|
code · table · image · audio · video · sensor ·
|
||||||
|
spatial/world states · tool_action · mixed · hidden_channel
|
||||||
|
```
|
||||||
|
|
||||||
|
Unsupported carriers in v1 → `reason="unsupported_carrier"`,
|
||||||
|
explicit fail or skip.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Fixture schema
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-<sub>-NNN",
|
||||||
|
"battery": "5f",
|
||||||
|
"sub_battery": "function",
|
||||||
|
"version": "v1",
|
||||||
|
|
||||||
|
"carrier": "claim_lattice",
|
||||||
|
"domain": "qa_answer",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"loss_report_refs": [],
|
||||||
|
"modality_notes": "Phase 1a uses text/claim-lattice; future carriers may include image, code, tool traces, world state.",
|
||||||
|
|
||||||
|
"task": { ... }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Per-sub-battery design
|
||||||
|
|
||||||
|
### 5.1 Function
|
||||||
|
|
||||||
|
**Definition:** does the system actually perform the designated
|
||||||
|
operation? Phase 1a: directive → expected output shape →
|
||||||
|
deterministic predicate.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-function-001",
|
||||||
|
"carrier": "claim_lattice",
|
||||||
|
"domain": "qa_answer",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"directive": "Return exactly two claim-lattice claims with evidence pointers.",
|
||||||
|
"answer_mode": "claim_lattice",
|
||||||
|
"expected_shape": {
|
||||||
|
"claim_count": 2,
|
||||||
|
"pointers_required": true
|
||||||
|
},
|
||||||
|
"evaluator": "shape_match",
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**v1 evaluators (deterministic only):**
|
||||||
|
|
||||||
|
```
|
||||||
|
shape_match · pointer_set_match · threshold_on_metric
|
||||||
|
```
|
||||||
|
|
||||||
|
**Future evaluators:** `ast_patch_applies`, `image_region_detected`,
|
||||||
|
`world_state_updated`, `tool_postcondition_met`.
|
||||||
|
|
||||||
|
**Metric:** `function_pass_rate = passed / total`.
|
||||||
|
|
||||||
|
### 5.2 Finetuning
|
||||||
|
|
||||||
|
**Definition:** measured improvement from parent state to child
|
||||||
|
state after adaptation. Adaptation includes SelfModel update, policy
|
||||||
|
update, benchmark-selected patch, π* canonicalizer update,
|
||||||
|
memory-root update, model-profile change, actual model finetune.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-finetuning-001",
|
||||||
|
"carrier": "selfmodel_snapshot",
|
||||||
|
"domain": "capability_transition",
|
||||||
|
"pi_star_ref": "pi_selfmodel_v1",
|
||||||
|
"parent_selfmodel": "SM_PARENT",
|
||||||
|
"child_selfmodel": "SM_CHILD",
|
||||||
|
"target_capability": "CAP-5S-SYLLOGISM",
|
||||||
|
"parent_measured_value": 0.45,
|
||||||
|
"child_measured_value": 0.65,
|
||||||
|
"expected_improvement_min": 0.05,
|
||||||
|
"resource_budget": {
|
||||||
|
"max_compute_ms_delta": 1000,
|
||||||
|
"max_storage_delta_bytes": 1000000
|
||||||
|
},
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Metrics:**
|
||||||
|
|
||||||
|
```
|
||||||
|
adaptation_improvement_rate
|
||||||
|
adaptation_efficiency = improvement_delta / capital_cost_delta
|
||||||
|
```
|
||||||
|
|
||||||
|
`adaptation_efficiency` hooks into the capital ledger (#000020,
|
||||||
|
landed) and prepares fitness-cost weighing for v8 fork choice
|
||||||
|
(#000012).
|
||||||
|
|
||||||
|
### 5.3 Falsification
|
||||||
|
|
||||||
|
**Definition:** does the system detect and correct errors?
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-falsification-001",
|
||||||
|
"carrier": "providence_record",
|
||||||
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"record": "planted_bad_providence_record",
|
||||||
|
"expected_reason": "WARRANT_MISSING",
|
||||||
|
"verifier_method_root": "...",
|
||||||
|
"repair_expected": true,
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Metrics:** `error_detection_rate`, `repair_success_rate`.
|
||||||
|
|
||||||
|
**Verifier-method-root tag:** every Falsification fixture pins the
|
||||||
|
`verifier_method_root` it was authored against. If the verifier
|
||||||
|
shape changes upstream, the bench reports a *mismatch warning*
|
||||||
|
instead of treating the fixture as a false failure.
|
||||||
|
|
||||||
|
**Hidden / alternate-media safety framing:** future carriers
|
||||||
|
(`carrier: "image"` / `"audio"` / `"hidden_channel"`) enable
|
||||||
|
defensive Falsification tasks — *detect unexpected carrier*,
|
||||||
|
*flag projection-loss risk*, *flag hidden-channel suspicion*,
|
||||||
|
*reject unsupported semantics*. Never generation, never
|
||||||
|
concealment.
|
||||||
|
|
||||||
|
### 5.4 Formulate
|
||||||
|
|
||||||
|
**Definition:** turn unstructured input into coherent, interpretable
|
||||||
|
structure.
|
||||||
|
|
||||||
|
Phase 1a: free text → claim lattice.
|
||||||
|
|
||||||
|
Future: image → scene graph; table → relational model; code →
|
||||||
|
AST/dependency-graph; sensor stream → event graph; world state →
|
||||||
|
object-relation graph; mixed media → cross-modal claim lattice.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-formulate-001",
|
||||||
|
"carrier": "text",
|
||||||
|
"domain": "claim_lattice",
|
||||||
|
"pi_star_ref": "claim-lattice@v1",
|
||||||
|
"input_text": "Document sentence with two claims and two pointers.",
|
||||||
|
"expected_lattice": {
|
||||||
|
"claim_count": 2,
|
||||||
|
"ordered": true,
|
||||||
|
"pointer_id_sets": [["E1"], ["E2"]]
|
||||||
|
},
|
||||||
|
"match_policy": {
|
||||||
|
"claim_text": "sorted_approx",
|
||||||
|
"pointer_ids": "exact_set",
|
||||||
|
"order": "required"
|
||||||
|
},
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Metric:** `structural_match_rate`.
|
||||||
|
|
||||||
|
**Match policy:** structural (sorted-approx claim text +
|
||||||
|
exact-set pointer IDs + required order), NOT exact-string. Avoids
|
||||||
|
brittleness without sacrificing determinism.
|
||||||
|
|
||||||
|
### 5.5 Feedback Loop
|
||||||
|
|
||||||
|
**Definition:** do observations at `t` change downstream
|
||||||
|
behavior/state at `t+1`?
|
||||||
|
|
||||||
|
This is the most important 5F component for recursive
|
||||||
|
self-improvement. Without Feedback Loop, the state-space exists
|
||||||
|
but does not learn.
|
||||||
|
|
||||||
|
**Fixture shape:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "5f-feedback-loop-001",
|
||||||
|
"carrier": "memory_snapshot",
|
||||||
|
"domain": "memory_root",
|
||||||
|
"pi_star_ref": "pi_memory_v1",
|
||||||
|
"chain": [
|
||||||
|
{"operation": "snapshot", "observation": "claim X is stale"},
|
||||||
|
{"operation": "update_memory", "observation": "memory branch marks X stale"},
|
||||||
|
{"operation": "snapshot", "expected_delta": "X appears in stale set"}
|
||||||
|
],
|
||||||
|
"expected": "pass"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Metrics:**
|
||||||
|
|
||||||
|
```
|
||||||
|
integration_coverage_rate
|
||||||
|
feedback_latency
|
||||||
|
downstream_effect_rate
|
||||||
|
feedback_efficiency = downstream_effect_count / capital_cost_delta
|
||||||
|
```
|
||||||
|
|
||||||
|
`feedback_efficiency` again hooks the capital ledger (#000020).
|
||||||
|
|
||||||
|
**Phase 1a uses existing surfaces only:** `memory_records`,
|
||||||
|
`memory_branch_summaries`, `audit_events`. No new substrate work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Required file changes
|
||||||
|
|
||||||
|
### 6.1 Module + fixtures
|
||||||
|
|
||||||
|
```
|
||||||
|
bench/batteries/b_5f.py # new
|
||||||
|
bench/fixtures/5f/function-v1.jsonl # ~10 seed (Phase 1a)
|
||||||
|
bench/fixtures/5f/finetuning-v1.jsonl # ~10 seed
|
||||||
|
bench/fixtures/5f/falsification-v1.jsonl # ~10 seed
|
||||||
|
bench/fixtures/5f/formulate-v1.jsonl # ~10 seed
|
||||||
|
bench/fixtures/5f/feedback-loop-v1.jsonl # ~10 seed
|
||||||
|
```
|
||||||
|
|
||||||
|
Each fixture file carries the standard `_meta` head with
|
||||||
|
`battery="5f"`.
|
||||||
|
|
||||||
|
### 6.2 Runner registration
|
||||||
|
|
||||||
In `bench/batteries/runner.py`:
|
In `bench/batteries/runner.py`:
|
||||||
|
|
||||||
|
|
@ -196,79 +382,147 @@ _BATTERIES = {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Existing `--all` mode picks up the new fixtures automatically once
|
### 6.3 Make targets
|
||||||
they're in `_DEFAULT_FIXTURES`.
|
|
||||||
|
|
||||||
### 4.3 Makefile additions
|
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
bench-5f: bootstrap ## 5F battery (Function/Finetuning/Falsification/Formulate/Feedback Loop)
|
bench-5f-function:
|
||||||
$(PY) -m bench.batteries.runner --battery 5f --sub function --fixtures bench/fixtures/5f/function-v1.jsonl
|
$(PY) -m bench.batteries.runner --battery 5f --sub function \
|
||||||
$(PY) -m bench.batteries.runner --battery 5f --sub finetuning --fixtures bench/fixtures/5f/finetuning-v1.jsonl
|
--fixtures bench/fixtures/5f/function-v1.jsonl
|
||||||
$(PY) -m bench.batteries.runner --battery 5f --sub falsification --fixtures bench/fixtures/5f/falsification-v1.jsonl
|
|
||||||
$(PY) -m bench.batteries.runner --battery 5f --sub formulate --fixtures bench/fixtures/5f/formulate-v1.jsonl
|
|
||||||
$(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop --fixtures bench/fixtures/5f/feedback-loop-v1.jsonl
|
|
||||||
|
|
||||||
bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite
|
bench-5f-finetuning:
|
||||||
|
$(PY) -m bench.batteries.runner --battery 5f --sub finetuning \
|
||||||
|
--fixtures bench/fixtures/5f/finetuning-v1.jsonl
|
||||||
|
|
||||||
|
bench-5f-falsification:
|
||||||
|
$(PY) -m bench.batteries.runner --battery 5f --sub falsification \
|
||||||
|
--fixtures bench/fixtures/5f/falsification-v1.jsonl
|
||||||
|
|
||||||
|
bench-5f-formulate:
|
||||||
|
$(PY) -m bench.batteries.runner --battery 5f --sub formulate \
|
||||||
|
--fixtures bench/fixtures/5f/formulate-v1.jsonl
|
||||||
|
|
||||||
|
bench-5f-feedback-loop:
|
||||||
|
$(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop \
|
||||||
|
--fixtures bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||||
|
|
||||||
|
bench-5f: bench-5f-function bench-5f-finetuning \
|
||||||
|
bench-5f-falsification bench-5f-formulate \
|
||||||
|
bench-5f-feedback-loop
|
||||||
|
|
||||||
|
bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite
|
||||||
```
|
```
|
||||||
|
|
||||||
### 4.4 Tests
|
`make bench-5s5t5f` becomes the canonical Dav1DPrometheus
|
||||||
|
non-embodied AGI fitness surface.
|
||||||
- `tests/test_bench_batteries.py` extension: positive coverage
|
|
||||||
for each new sub-battery.
|
|
||||||
- Determinism tests: each runner produces identical
|
|
||||||
`BatteryResult.fixture_digest` and `runtime_digest` across
|
|
||||||
back-to-back runs.
|
|
||||||
|
|
||||||
### 4.5 Bumps
|
|
||||||
|
|
||||||
- New module `bench/batteries/b_5f.py`.
|
|
||||||
- 5 new fixture files.
|
|
||||||
- New Makefile targets.
|
|
||||||
- No schema changes; no policy hash changes.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Out of scope
|
## 7. Tests
|
||||||
|
|
||||||
- Phase 1b fixture expansion (target 30-50 per sub-battery). Phase
|
```
|
||||||
1a's seed set lands first.
|
test_5f_function_runs
|
||||||
- Cross-battery composition (e.g., "score this agent across
|
test_5f_finetuning_runs
|
||||||
5S+5T+5F"). Aggregation is its own concern.
|
test_5f_falsification_runs
|
||||||
- Calibration of metric thresholds. Phase 1a measures; thresholds
|
test_5f_formulate_runs
|
||||||
for v8 selection acceptance come later.
|
test_5f_feedback_loop_runs
|
||||||
- Auto-grading via LLM. All 5F evaluators are deterministic
|
test_5f_fixture_digest_stable
|
||||||
predicates; LLM-as-judge is future work.
|
test_5f_runtime_digest_stable
|
||||||
|
test_5f_fixture_accepts_carrier_metadata
|
||||||
|
test_5f_unsupported_future_carrier_fails_cleanly
|
||||||
|
```
|
||||||
|
|
||||||
|
Sub-battery-specific:
|
||||||
|
|
||||||
|
```
|
||||||
|
test_5f_function_shape_match_evaluator
|
||||||
|
test_5f_function_pointer_set_match_evaluator
|
||||||
|
test_5f_function_threshold_on_metric_evaluator
|
||||||
|
test_5f_finetuning_requires_parent_and_child_selfmodel
|
||||||
|
test_5f_falsification_verifier_method_root_mismatch_warns
|
||||||
|
test_5f_formulate_pointer_id_set_exact_match
|
||||||
|
test_5f_feedback_loop_observation_affects_downstream_snapshot
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Risks & open questions
|
## 8. Out of scope
|
||||||
|
|
||||||
- **Function evaluator design.** "Did the system execute the
|
- Phase 1b fixture expansion (target 30-50 per sub-battery).
|
||||||
designated task" requires a per-task predicate. v1 ships a
|
- Cross-battery composite scoring (5S + 5T + 5F aggregation is its
|
||||||
small set of canonical evaluator types (shape-match,
|
own concern).
|
||||||
pointer-set-match, threshold-on-metric). New evaluator types
|
- Calibration of metric thresholds for v8 selection acceptance
|
||||||
land in follow-ups.
|
(depends on #000012 v8 paper).
|
||||||
- **Finetuning needs SelfModel chains.** Realistic fixtures
|
- LLM-as-judge for any 5F task (forbidden by hard constraint).
|
||||||
require real shard data with parent→child SelfModel transitions.
|
- Multimodal Phase 1a (text/claim-lattice/memory only; multimodal
|
||||||
Phase 1a uses synthetic chains generated in-test; Phase 1b
|
comes after #000013 + #000015 follow-ups land).
|
||||||
pulls from production shards.
|
- Hidden-channel **generation**. Defensive detection only, in
|
||||||
- **Falsification fixtures are fragile.** A planted-error fixture
|
Phase 1b+.
|
||||||
bound to a specific verifier signature may stop catching the
|
|
||||||
error if the verifier changes shape. Mitigation: tag each
|
|
||||||
Falsification fixture with the `verifier_method_root` it was
|
|
||||||
authored against; bench reports flag mismatches.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Status
|
## 9. Risks & mitigations
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|---|---|
|
||||||
|
| Function evaluator vague | v1 allows only named deterministic evaluator types: `shape_match`, `pointer_set_match`, `threshold_on_metric`. Future evaluators land in follow-ups. |
|
||||||
|
| Finetuning fakes improvement | Require parent + child SelfModel roots; require measured_value pre/post; require improvement threshold; record capital cost. |
|
||||||
|
| Falsification fixtures stale | Tag every fixture with `verifier_method_root`. Bench reports mismatch instead of false failure. |
|
||||||
|
| Formulate exact-string brittle | Structural match only: claim count + order + pointer-ID-set + claim-text approximate/sorted. |
|
||||||
|
| Feedback Loop needs substrate work | Use existing `memory_records`, `memory_branch_summaries`, `audit_events` surfaces. Phase 1a explicitly forbids new substrate. |
|
||||||
|
| Premature multimodal scope creep | Carrier metadata in fixtures; non-text carriers fail/skip explicitly. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Closure criterion
|
||||||
|
|
||||||
|
### Phase 1a closes when:
|
||||||
|
|
||||||
|
1. `bench/batteries/b_5f.py` lands.
|
||||||
|
2. All five sub-battery runners are non-stub.
|
||||||
|
3. Each sub-battery has ≥10 deterministic seed fixtures.
|
||||||
|
4. Fixture metadata includes `carrier` / `domain` / `pi_star_ref`
|
||||||
|
/ `loss_report_refs` / `modality_notes` fields.
|
||||||
|
5. `make bench-5f` passes end-to-end.
|
||||||
|
6. `make bench-5s5t5f` exists and runs the complete Dav1DPrometheus
|
||||||
|
suite.
|
||||||
|
7. Runtime + fixture digests are deterministic across repeated
|
||||||
|
runs.
|
||||||
|
8. No schema, governance hash, or canonicalization-version changes.
|
||||||
|
9. No LLM-as-judge anywhere in 5F runners.
|
||||||
|
|
||||||
|
### Phase 1b keeps the ticket in-progress until:
|
||||||
|
|
||||||
|
10. 30-50 fixtures per sub-battery.
|
||||||
|
11. Real shard-based Finetuning chains (replacing synthetic
|
||||||
|
parent→child SelfModel pairs from Phase 1a).
|
||||||
|
12. Expanded Falsification planted-error suite covering every
|
||||||
|
documented failure-motif tag.
|
||||||
|
13. Feedback Loop latency / efficiency metrics calibrated against
|
||||||
|
real workload data.
|
||||||
|
14. Threshold calibration for v8 selection acceptance handed off
|
||||||
|
to ticket #000012.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Status
|
||||||
|
|
||||||
**Open · awaiting go/no-go.** Larger surface than #000023 / #000024
|
**Open · awaiting go/no-go.** Larger surface than #000023 / #000024
|
||||||
because 5F has five sub-batteries vs three remaining in 5S and
|
(five sub-batteries vs three / four remaining). Recommended
|
||||||
four in 5T. Recommended sequencing: ship Phase 1a in one push
|
sequencing: ship Phase 1a in one push (seed fixtures + non-stub
|
||||||
(seed-fixture per sub-battery, ~10 each), then Phase 1b expands.
|
runners), Phase 1b expands.
|
||||||
|
|
||||||
Closure criterion (Phase 1a): `bench/batteries/b_5f.py` lands
|
The synthesis statement that justifies 5F's existence as a distinct
|
||||||
with non-stub runners for all five sub-batteries; seed fixtures
|
axis:
|
||||||
+ Makefile targets pass `make bench-5f` end-to-end. Phase 1b
|
|
||||||
keeps the ticket open in-progress until fixtures are expanded.
|
> **SQD gives the invariant projection and verifier model.
|
||||||
|
> Merkle-AGI v7 gives multimodal committed DAG structure. 5S gives
|
||||||
|
> sign/meaning/derivation fitness. 5T gives transfer, logic,
|
||||||
|
> triangulation, and time. 5F gives operation, adaptation,
|
||||||
|
> falsification, formulation, and feedback. Together they
|
||||||
|
> instantiate an explicit discrete state-space/time in which
|
||||||
|
> counting, mathematics, and logic become auditable operations over
|
||||||
|
> committed state — not just text generated by a latent model.**
|
||||||
|
|
||||||
|
5F is the operational layer that turns the substrate into an
|
||||||
|
adaptive learning organism. Without it, the state-space exists
|
||||||
|
but does not improve.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue