docs/tickets: open #000023-#000025 — Dav1DPrometheus 5S/5T/5F coverage
Three new design-only tickets surfacing the gaps between arborist's current bench harness (ticket #000021 Phase 1a, landed) and Dav1DPrometheus's authoritative 5S/5F/5T evaluation framework. - #000023 — 5S Phase 1b: real implementations + fixtures for Syllogism, Synthesis, Semiotics (currently stubbed). - #000024 — 5T Phase 1b: rename Transfer→Transfer Learning, Truth→Truthtables, Timing→Time to honor Dav1DPrometheus's vocabulary; ship real Triangulation, Truthtables, Transitivity, Time runners (currently stubbed). Time integrates with memory_root (#000017) for the first measurable use of v8 substrate as fitness target. - #000025 — 5F battery: entirely new — Function, Finetuning, Falsification, Formulate, Feedback Loop. arborist had no 5F coverage before this ticket; the SQD whitepaper omitted the axis. Each sub-battery integrates with surfaces already shipped (selfmodel_records, providence_cache.falsification_state, memory_branch_summaries). Source attribution: Legally Unprecedented Dav1DPrometheus (BasementAGI host). Honoring his framework as the authoritative taxonomy for non-embodied AGI evaluation. Next ID bumped 000023 → 000026.
This commit is contained in:
parent
a64d941528
commit
10d2db621c
4 changed files with 746 additions and 1 deletions
|
|
@ -57,6 +57,9 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000025 | 5F battery (Function · Finetuning · Falsification · Formulate · Feedback Loop) | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000024 | 5T Phase 1b + Dav1DPrometheus vocabulary alignment | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000023 | 5S Phase 1b: Syllogism · Synthesis · Semiotics | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000022 | Adapter LossReport (PRD I9 analogue) | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000021 | 5S/5T/5R benchmark fixtures + harness | in progress · Phase 1a landed 2026-05-07 | 2026-05-07 | — |
|
||||
| #000020 | Capital-cost ledger (8-capital queues) | closed · landed 2026-05-07 | 2026-05-07 | — |
|
||||
|
|
@ -82,4 +85,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000023`
|
||||
`000026`
|
||||
|
|
|
|||
|
|
@ -0,0 +1,230 @@
|
|||
# Ticket #000023 — 5S Phase 1b: Syllogism, Synthesis, Semiotics
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Complete the 5S battery's three remaining sub-batteries —
|
||||
Syllogism, Synthesis, Semiotics — that landed as zero-task stubs in
|
||||
ticket #000021 Phase 1a. Each sub-battery gets a real fixture set
|
||||
(seed → expand) and a runner that exercises arborist's actual
|
||||
surface (claim-lattice parser, retrieval pipeline, label-swap
|
||||
behavior).
|
||||
**Audience:** fox + future bench authors + downstream v8 / v7-W work
|
||||
that needs concrete fitness targets across all 5S dimensions.
|
||||
**Hard constraint:** fixtures land deterministically (committed
|
||||
JSONL with version pin). Bench-maxing discipline (5pp signal floor,
|
||||
n≥3) applies. Each sub-battery is independently runnable via
|
||||
`make bench-5s-<sub>`. Adding sub-batteries does NOT alter Phase 1a
|
||||
behavior — Syntax + Semantics keep their existing fixture digests.
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) named the 5Ss in his
|
||||
authoritative framework; arborist honors the original wording.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Ticket #000021 Phase 1a shipped runners + seed fixtures for 5S
|
||||
Syntax (`parse-pass`) and 5S Semantics (`equivalence`). The other
|
||||
three sub-batteries — Syllogism, Synthesis, Semiotics — landed as
|
||||
stubs that return zero-task `BatteryResult` objects so the full
|
||||
suite at least runs. Stubs are visible in `bench/batteries/b_5s.py`.
|
||||
|
||||
Without real implementations:
|
||||
|
||||
| Gap | Failure mode |
|
||||
|---|---|
|
||||
| **Syllogism** | No way to bench arborist's deductive-reasoning surface. v8 selection + SelfModel capability claims have no measurable target for "stepwise entailment validity." |
|
||||
| **Synthesis** | No bench for "given a goal + a fact set, can the system produce a passing derivation?" — directly the claim-lattice + retrieval pipeline's output, which today only gets ad-hoc QA bench coverage. |
|
||||
| **Semiotics** | Label-swap invariance under controlled symbol changes is unmeasured. Drift in proper-noun handling, code-name conventions, or alphabet shifts goes undetected. |
|
||||
|
||||
### 1.1 Sub-battery semantics (per ticket #000021 §4.2)
|
||||
|
||||
- **Syllogism:** each task is a proof chain + a step index +
|
||||
expected step-valid pass/fail. Metric: step-validity-rate.
|
||||
- **Synthesis:** each task is a goal description + a fact set +
|
||||
expected derivation pass/fail. Metric: derivation-pass-rate.
|
||||
- **Semiotics:** each task is a controlled label-swap + invariant
|
||||
expected. Metric: invariance-under-swap.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
|
||||
### 2.1 Fixture sourcing for Syllogism
|
||||
|
||||
**A. Hand-curated proof chains drawn from existing claim-lattice tests.**
|
||||
arborist's `tests/test_claim_lattice.py` and `tests/test_directives.py`
|
||||
already contain stepwise-claim examples. Promoting them to
|
||||
fixtures gets us 30-50 entries cheaply.
|
||||
|
||||
**B. Synthetic generators (template-based syllogisms).** Cheap;
|
||||
templates need to be committed for reproducibility.
|
||||
|
||||
**Recommendation:** A first; add B in Phase 2 once the harness
|
||||
stabilizes.
|
||||
|
||||
### 2.2 Fixture sourcing for Synthesis
|
||||
|
||||
**A. Ablate from existing QA fixtures.** Take questions from
|
||||
`bench/qa_sweep.py` corpora; record which sources retrieve as the
|
||||
fact set; mark expected derivation status.
|
||||
|
||||
**B. Compose from claim-lattice docs.** Use docs already in shards;
|
||||
build (goal, fact_set, derivation_expected) tuples by stripping
|
||||
claims and asking whether they should re-derive.
|
||||
|
||||
**Recommendation:** A — anchors against existing QA bench so
|
||||
Synthesis-rate trends correlate with overall QA rate.
|
||||
|
||||
### 2.3 Fixture sourcing for Semiotics
|
||||
|
||||
**A. Hand-curated label-swap pairs.** Examples: `"AMD"↔"Advanced Micro
|
||||
Devices"`, `"NYC"↔"New York City"`, code-name swaps inside
|
||||
identical text.
|
||||
|
||||
**B. Generated swaps via the concept-relations table.** arborist's
|
||||
`concept_relations` already has synonym pairs from corpus
|
||||
extraction; promote them as semiotic swap fixtures.
|
||||
|
||||
**Recommendation:** B — leverages existing per-shard data, makes
|
||||
fixtures regenerable from the corpus rather than hand-pinned.
|
||||
|
||||
### 2.4 Runner integration
|
||||
|
||||
Each sub-battery is a function in `bench/batteries/b_5s.py`
|
||||
matching the existing `run_syntax`/`run_semantics` signature.
|
||||
Replace the stub `_stub_result` returns with real implementations.
|
||||
Existing test in `tests/test_bench_batteries.py
|
||||
::test_stub_sub_batteries_return_zero` will need to flip to
|
||||
non-zero assertions once these land.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
Land in this order:
|
||||
|
||||
1. **Syllogism** first — purely lexical (claim-lattice traversal),
|
||||
no shard reads, fastest implementation.
|
||||
2. **Semiotics** second — depends on `concept_relations` lookup
|
||||
but no LLM call; deterministic.
|
||||
3. **Synthesis** third — requires shard reads + retrieval pipeline;
|
||||
slowest, biggest surface, highest variance. Defer until
|
||||
Syllogism + Semiotics stabilize the harness.
|
||||
|
||||
Phase 1b closure = all three sub-batteries shipping with at minimum
|
||||
**30 deterministic fixtures each** and runners that integrate
|
||||
cleanly into `make bench-5s`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Fixture files
|
||||
|
||||
```
|
||||
bench/fixtures/5s/syllogism-v1.jsonl # 30+ tasks
|
||||
bench/fixtures/5s/synthesis-v1.jsonl # 30+ tasks
|
||||
bench/fixtures/5s/semiotics-v1.jsonl # 30+ tasks
|
||||
```
|
||||
|
||||
Each fixture file carries the standard `_meta` head:
|
||||
|
||||
```json
|
||||
{"_meta":{"battery":"5s","sub_battery":"syllogism","version":"v1","task_count":30}}
|
||||
```
|
||||
|
||||
### 4.2 Runner additions
|
||||
|
||||
In `bench/batteries/b_5s.py`, replace stub bodies:
|
||||
|
||||
```python
|
||||
def run_syllogism(fixtures_path: Path) -> BatteryResult:
|
||||
# for each task: parse claim chain, walk step_index, assert
|
||||
# the claim references valid evidence + the prior step entails
|
||||
# this one under claim-lattice semantics. Metric:
|
||||
# step_validity_rate = pass_count / (pass + fail).
|
||||
...
|
||||
```
|
||||
|
||||
Same shape for `run_synthesis`, `run_semiotics`. Existing
|
||||
`_runtime_digest` helper covers the runtime fingerprint.
|
||||
|
||||
### 4.3 Makefile additions
|
||||
|
||||
```makefile
|
||||
bench-5s-syllogism: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub syllogism \
|
||||
--fixtures bench/fixtures/5s/syllogism-v1.jsonl
|
||||
|
||||
bench-5s-synthesis: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub synthesis \
|
||||
--fixtures bench/fixtures/5s/synthesis-v1.jsonl
|
||||
|
||||
bench-5s-semiotics: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5s --sub semiotics \
|
||||
--fixtures bench/fixtures/5s/semiotics-v1.jsonl
|
||||
```
|
||||
|
||||
Existing `bench-5s` aggregate target picks them up via `--all`
|
||||
once the default fixture map in `runner.py` includes them.
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- Replace `test_stub_sub_batteries_return_zero` parametrizations
|
||||
for the three sub-batteries with positive coverage:
|
||||
`test_5s_syllogism_runs`, `test_5s_synthesis_runs`,
|
||||
`test_5s_semiotics_runs`.
|
||||
- Add fixture-digest stability test for each new fixture file.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New fixture files (additive). Phase 1a fixture digests stay
|
||||
pinned.
|
||||
- Runner-stub replacements in `b_5s.py`.
|
||||
- New Makefile targets.
|
||||
- No schema changes. No `governance_policy_hash` changes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- 5T Phase 1b (Triangulation, Truthtables, Transitivity, Time —
|
||||
ticket #000024).
|
||||
- 5F battery (Function, Finetuning, Falsification, Formulate,
|
||||
Feedback Loop — ticket #000025).
|
||||
- 5R battery (workspace operators) — depends on SelfModel
|
||||
(#000014, landed) + memory_root (#000017, landed); reserved as
|
||||
Phase 2 of #000021.
|
||||
- Cross-corpus expansion (Phase 3 of #000021).
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
|
||||
- **Syllogism fixture quality.** Hand-curated chains can over-fit
|
||||
to arborist's current claim-lattice parser. Mitigation: pull
|
||||
from at least three different test files so coverage spans
|
||||
multiple authoring styles.
|
||||
- **Synthesis variance.** Depends on retrieval which has
|
||||
per-shard variance. Mitigation: pin shard set in the fixture
|
||||
meta; bench-maxing 5pp signal floor accounts for noise.
|
||||
- **Semiotics blast radius.** A bad swap fixture could mask
|
||||
genuine drift if the swap is too aggressive (e.g., swaps that
|
||||
also change semantic meaning, not just surface). Mitigation:
|
||||
curate from `concept_relations` of kind `synonym` only — those
|
||||
are corpus-derived equivalences with known semantic preservation.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Three sub-batteries × ~30 fixtures
|
||||
× one runner each ≈ 600-800 lines of code + fixtures. Estimated
|
||||
mid-sized session.
|
||||
|
||||
Closure criterion: all three runners non-stub, fixture files
|
||||
landed with version pins, `make bench-5s` runs all five 5S
|
||||
sub-batteries (Syntax, Semantics, Syllogism, Synthesis, Semiotics)
|
||||
and emits a JSON result per sub-battery. Existing Phase 1a
|
||||
fixtures keep their digests.
|
||||
238
docs/tickets/ticket-000024-5t-phase-1b-and-rename.md
Normal file
238
docs/tickets/ticket-000024-5t-phase-1b-and-rename.md
Normal file
|
|
@ -0,0 +1,238 @@
|
|||
# Ticket #000024 — 5T Phase 1b: complete sub-batteries + align vocabulary with Dav1DPrometheus
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Complete the 5T battery — rename existing sub-batteries
|
||||
to match Dav1DPrometheus's authoritative vocabulary
|
||||
(Transfer → Transfer Learning, Truth → Truthtables, Timing →
|
||||
Time), and ship real implementations + fixtures for the four
|
||||
remaining stub sub-batteries (Triangulation, Truthtables,
|
||||
Transitivity, Time).
|
||||
**Audience:** fox + future bench authors + anyone composing a v8
|
||||
fitness target across the 5T axes.
|
||||
**Hard constraint:** the rename is a NEW fixture-version landing
|
||||
(`v1` → `v2`); existing `transfer-v1.jsonl` stays in place under
|
||||
its sub-battery name `transfer` so #000021 Phase 1a digests don't
|
||||
shift retroactively. New work uses the renamed identifiers.
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Ts: Transitivity,
|
||||
Transfer Learning, Triangulation, Truthtables, Time."*
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Two coupled gaps:
|
||||
|
||||
### 1.1 Vocabulary drift from authoritative framework
|
||||
|
||||
The SQD whitepaper used: **Transfer · Triangulate · Timing ·
|
||||
Transitivity · Truth.**
|
||||
|
||||
Dav1DPrometheus's published framework uses: **Transitivity ·
|
||||
**Transfer Learning** · Triangulation · **Truthtables** · **Time**.**
|
||||
|
||||
Three differences:
|
||||
|
||||
| arborist (current) | Dav1DPrometheus | Why his is right |
|
||||
|---|---|---|
|
||||
| `transfer` | `transfer-learning` | "Transfer" alone is ambiguous; "Transfer Learning" pins the concrete ML primitive. |
|
||||
| `truth` | `truthtables` | "Truth" is philosophical; "Truthtables" names the concrete logical-scenario coverage primitive. |
|
||||
| `timing` | `time` | "Timing" implies wall-clock; "Time" implies temporal-context maintenance, which is the actual semantic. |
|
||||
|
||||
arborist landed Phase 1a with the SQD-whitepaper names. Per memory
|
||||
note `dav1dprometheus_framework.md`, his wording is authoritative —
|
||||
arborist honors it.
|
||||
|
||||
### 1.2 Four stub sub-batteries return zero tasks
|
||||
|
||||
Same as 5S (ticket #000023): Triangulation, Truthtables,
|
||||
Transitivity, Time all currently return zero-task `BatteryResult`
|
||||
objects via `_stub_result` in `bench/batteries/b_5t.py`. Without
|
||||
real implementations:
|
||||
|
||||
| Sub-battery | Failure mode under stubs |
|
||||
|---|---|
|
||||
| **Triangulation** | No bench for "do independent verifiers agree?" — directly the v7 §11.4 cross-source consistency claim. |
|
||||
| **Truthtables** | No coverage of "for these N propositional variables, does the system handle all 2^N cases?" — gates v8 selection's ability to certify logical-completeness claims. |
|
||||
| **Transitivity** | No bench for "if A→B and B→C, does the system derive A→C?" — multi-step entailment is a SelfModel capability claim with no current measurement. |
|
||||
| **Time** | No bench for "does the system maintain temporal context across queries?" — directly relates to memory_root + SelfModel's lineage chain, both shipped but unbenchmarked. |
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
|
||||
### 2.1 Rename strategy
|
||||
|
||||
**A. Hard rename in place.** Update `b_5t.py` function names and
|
||||
fixture filenames; existing Phase 1a digests change. Cleanest
|
||||
forward but invalidates Phase 1a results.
|
||||
|
||||
**B. Coexist v1 + v2.** Keep `transfer-v1.jsonl` + `run_transfer`
|
||||
function in place; add `transfer-learning-v2.jsonl` +
|
||||
`run_transfer_learning`. New work uses v2 names. Old `bench-5t`
|
||||
target still runs against v1.
|
||||
|
||||
**C. Soft rename (alias).** Keep file names; add alias mapping in
|
||||
the runner so `--sub transfer-learning` resolves to `run_transfer`.
|
||||
Code stays ugly forever.
|
||||
|
||||
**Recommendation:** B. Phase 1a was honest at the time it landed
|
||||
(SQD whitepaper was the only source then); we don't retroactively
|
||||
break it. New work explicitly uses Dav1DPrometheus's vocabulary.
|
||||
|
||||
### 2.2 Fixture sourcing
|
||||
|
||||
- **Triangulation** — replay arborist's existing four-strategy
|
||||
verifier (quote/span/entity/paraphrase) on a fixture set;
|
||||
metric is the per-task agreement rate. Already-shipped verifier
|
||||
surface; fixtures are easy.
|
||||
- **Truthtables** — small-N propositional fixtures (N=2..4); each
|
||||
fixture lists all 2^N input combinations + the expected output;
|
||||
arborist's claim-lattice has to evaluate every row. New surface
|
||||
(currently arborist doesn't do propositional eval); requires a
|
||||
thin wrapper.
|
||||
- **Transitivity** — multi-step claim chains pulled from existing
|
||||
claim-lattice tests; metric is full-chain-pass-rate.
|
||||
- **Time** — multi-query sequences against the same shard set;
|
||||
metric is "does memory_root snapshot N+1 preserve information
|
||||
from snapshot N?" Direct integration with memory-root (#000017,
|
||||
landed).
|
||||
|
||||
### 2.3 Cross-reference to memory-root and SelfModel
|
||||
|
||||
The Time sub-battery implementation reads memory_records and
|
||||
selfmodel_records to assert temporal-context preservation. This
|
||||
makes #000024 the first ticket where the v8 substrate
|
||||
(SelfModel + memory_root) gets a measurable bench target, not
|
||||
just an audit-chain artifact.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
Two-phase landing:
|
||||
|
||||
**Phase 1b.1 — Rename + cosmetic alignment.** Add v2 fixture file
|
||||
names + new sub-battery aliases. Keep v1 in place. ~100 lines.
|
||||
|
||||
**Phase 1b.2 — Implement four sub-batteries.** Triangulation
|
||||
first (existing surface), then Transitivity (lexical), then
|
||||
Truthtables (new wrapper), then Time (memory-root-integrated).
|
||||
~600-800 lines.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Fixture files (new)
|
||||
|
||||
```
|
||||
bench/fixtures/5t/transfer-learning-v2.jsonl # rename of transfer-v1
|
||||
bench/fixtures/5t/triangulation-v1.jsonl
|
||||
bench/fixtures/5t/truthtables-v1.jsonl
|
||||
bench/fixtures/5t/transitivity-v1.jsonl
|
||||
bench/fixtures/5t/time-v1.jsonl
|
||||
```
|
||||
|
||||
`transfer-v1.jsonl` stays in place; `bench-5t-transfer` still runs
|
||||
it. Phase 1a tests stay green.
|
||||
|
||||
### 4.2 Runner additions
|
||||
|
||||
In `bench/batteries/b_5t.py`, real implementations replace stubs:
|
||||
|
||||
- `run_transfer_learning(fixtures_path)` — same logic as
|
||||
`run_transfer`; alias.
|
||||
- `run_triangulation(fixtures_path)` — for each task, call
|
||||
arborist's four verifier strategies on (claim, evidence) pairs
|
||||
and report agreement rate.
|
||||
- `run_truthtables(fixtures_path)` — for each task, evaluate the
|
||||
claim-lattice over 2^N input combinations.
|
||||
- `run_transitivity(fixtures_path)` — chase A→B, B→C through the
|
||||
claim-lattice; assert C derives.
|
||||
- `run_time(fixtures_path)` — multi-query memory_root preservation;
|
||||
reads `memory_records` to assert continuity.
|
||||
|
||||
### 4.3 Makefile additions
|
||||
|
||||
```makefile
|
||||
bench-5t-transfer-learning: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5t --sub transfer-learning \
|
||||
--fixtures bench/fixtures/5t/transfer-learning-v2.jsonl
|
||||
|
||||
bench-5t-triangulation: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5t --sub triangulation \
|
||||
--fixtures bench/fixtures/5t/triangulation-v1.jsonl
|
||||
|
||||
bench-5t-truthtables: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5t --sub truthtables \
|
||||
--fixtures bench/fixtures/5t/truthtables-v1.jsonl
|
||||
|
||||
bench-5t-transitivity: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5t --sub transitivity \
|
||||
--fixtures bench/fixtures/5t/transitivity-v1.jsonl
|
||||
|
||||
bench-5t-time: bootstrap
|
||||
$(PY) -m bench.batteries.runner --battery 5t --sub time \
|
||||
--fixtures bench/fixtures/5t/time-v1.jsonl
|
||||
```
|
||||
|
||||
Existing `bench-5t` aggregate keeps `transfer` (v1) for back-compat.
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- Add `test_5t_transfer_learning_alias` — confirms new name resolves
|
||||
to same surface as `transfer`.
|
||||
- Add positive tests for each new sub-battery.
|
||||
- Update stub-test parametrization to remove the four sub-batteries
|
||||
that get real implementations.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New fixture files (additive).
|
||||
- New runner functions; old ones (`run_transfer`,
|
||||
`run_triangulate`, etc.) stay in place for back-compat.
|
||||
- New Makefile targets.
|
||||
- No schema changes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- 5S Phase 1b — ticket #000023.
|
||||
- 5F battery (entirely new) — ticket #000025.
|
||||
- 5R battery (workspace operators) — Phase 2 of #000021.
|
||||
- Retiring the v1 names. Once v2 is the standard, deprecate v1
|
||||
in a follow-up ticket; this ticket leaves both live.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
|
||||
- **Truthtables surface.** arborist doesn't currently do
|
||||
propositional evaluation; building the wrapper is real work.
|
||||
Defer if scope compresses; ship the other three first.
|
||||
- **Time integration depth.** memory_root snapshots are advisory
|
||||
(default policy off). The Time bench can still measure
|
||||
preservation across snapshots even when binding is off, but
|
||||
results may not reflect production behavior under different
|
||||
policies. Document the policy assumption in the fixture meta.
|
||||
- **Backward compatibility.** Old `bench-5t` (v1 names) stays
|
||||
alongside new `bench-5t-*` (v2 names). Risk of confusion;
|
||||
mitigate by clear naming + a `docs/bench-vocabulary.md` note
|
||||
if needed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Mid-to-large surface; biggest sub-
|
||||
battery (Time) needs memory-root integration. Recommended
|
||||
implementation sequence: rename first, then sub-batteries in the
|
||||
order Triangulation → Transitivity → Time → Truthtables.
|
||||
|
||||
Closure criterion: all five 5T sub-batteries (Transfer Learning,
|
||||
Triangulation, Truthtables, Transitivity, Time) have non-stub
|
||||
runners + fixture files; `make bench-5t` runs the full set under
|
||||
v2 names; v1 `bench-5t-transfer` still passes.
|
||||
274
docs/tickets/ticket-000025-5f-battery.md
Normal file
274
docs/tickets/ticket-000025-5f-battery.md
Normal file
|
|
@ -0,0 +1,274 @@
|
|||
# Ticket #000025 — 5F battery: Function, Finetuning, Falsification, Formulate, Feedback Loop
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Open the 5F battery — Dav1DPrometheus's third evaluation
|
||||
axis, missing entirely from arborist before this ticket. Each sub-
|
||||
battery measures a distinct dimension of agent quality:
|
||||
**Function** (task alignment), **Finetuning** (adaptation
|
||||
efficiency), **Falsification** (error detection + correction),
|
||||
**Formulate** (logical structuring), **Feedback Loop** (continuous
|
||||
improvement).
|
||||
**Audience:** fox + future bench authors. Once landed, 5F + 5S + 5T
|
||||
together form the complete non-embodied AGI evaluation surface
|
||||
Dav1DPrometheus published.
|
||||
**Hard constraint:** 5F integrates with surfaces that already exist
|
||||
in arborist — `providence_cache.falsification_state`, `audit_events`,
|
||||
`memory_records` (#000017), `selfmodel_records` (#000014). Don't
|
||||
require new substrate work to land 5F.
|
||||
|
||||
**Source:** Dav1DPrometheus (BasementAGI host) — *"5Fs: Function,
|
||||
Finetuning, Falsification, Formulate, Feedback Loop."* Honoring his
|
||||
legacy.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Dav1DPrometheus's framework names three orthogonal evaluation axes:
|
||||
|
||||
```text
|
||||
5Ss = linguistic precision (syntax + semantics + syllogism + synthesis + semiotics)
|
||||
5Ts = temporal/cross reasoning (transitivity + transfer learning + triangulation + truthtables + time)
|
||||
5Fs = operational quality (function + finetuning + falsification + formulate + feedback loop)
|
||||
```
|
||||
|
||||
arborist's bench harness (ticket #000021) implements 5S + 5T as
|
||||
fixture-driven batteries. **5F has no representation in arborist.**
|
||||
The original SQD whitepaper omitted it; Dav1DPrometheus's framework
|
||||
is the only source for it.
|
||||
|
||||
Without 5F:
|
||||
|
||||
| Sub-battery | Why arborist needs it |
|
||||
|---|---|
|
||||
| **Function** | Measures whether the system actually executes its designated task. arborist's QA layer has answer-mode bench coverage but not a clean "did the system DO what it was asked" metric. |
|
||||
| **Finetuning** | Measures speed/efficiency of adaptation. SelfModel parent→child transitions are unbenchmarked. |
|
||||
| **Falsification** | Error-detection accuracy is core to v9.8 admissibility, but `providence_cache.falsification_state` transitions are not benched. |
|
||||
| **Formulate** | Coherence of organized knowledge structures — directly the claim-lattice's job, currently only measured by per-claim verification rate, not chain-level coherence. |
|
||||
| **Feedback Loop** | Integration efficiency for incoming corrections. Memory-root branch projections (#000017) update on snapshot but speed of integration is unmeasured. |
|
||||
|
||||
### 1.1 Why this is a separate ticket from #000021
|
||||
|
||||
#000021 was scoped to the SQD whitepaper's 5S + 5T. Adding 5F
|
||||
would have changed scope mid-design. Treating 5F as its own
|
||||
ticket keeps each battery's story coherent and lets fox approve
|
||||
or punt 5F independently.
|
||||
|
||||
---
|
||||
|
||||
## 2. Sub-battery semantics (per Dav1DPrometheus)
|
||||
|
||||
### 2.1 Function
|
||||
|
||||
> *Task Alignment: Precision in executing designated tasks.
|
||||
> Operational Effectiveness: Empirical success rate in task execution.*
|
||||
|
||||
Each task: a directive (e.g., "summarize document X"), a target
|
||||
output shape (e.g., claim-lattice with N claims), and an
|
||||
evaluation predicate (does the produced output meet the shape?).
|
||||
Metric: function-pass-rate.
|
||||
|
||||
Maps to arborist surfaces: `arborist ask` invocations under
|
||||
specific `answer_mode` settings.
|
||||
|
||||
### 2.2 Finetuning
|
||||
|
||||
> *Performance Improvement: Quantitative percentage increase in
|
||||
> performance metrics post-finetuning.
|
||||
> Adaptability: Efficiency in adapting to new tasks, measured by
|
||||
> time and resources required.*
|
||||
|
||||
Each task: a parent SelfModel + a target capability claim that
|
||||
the parent fails + a child SelfModel that should pass. Metric:
|
||||
fraction of parent→child transitions where the child's measured
|
||||
metric improves over parent's.
|
||||
|
||||
Maps to arborist surfaces: `selfmodel_records` parent→child chain
|
||||
+ `selfmodel_capability_claims` measured_value tracking.
|
||||
|
||||
### 2.3 Falsification
|
||||
|
||||
> *Error Detection Accuracy: Rate of correctly identifying system
|
||||
> errors. Correction Effectiveness: Success rate in rectifying
|
||||
> identified errors.*
|
||||
|
||||
Each task: a known-bad providence record (planted) + the expected
|
||||
falsification reason. Metric: fraction of planted records the
|
||||
verifier correctly flags + fraction of repair operations that
|
||||
restore correctness.
|
||||
|
||||
Maps to arborist surfaces: `falsification_state` transitions,
|
||||
`providence_repair` audit events, `arborist.qa.repair`.
|
||||
|
||||
### 2.4 Formulate
|
||||
|
||||
> *Logical Structuring: Degree of coherence in organized
|
||||
> knowledge structures.
|
||||
> Framework Clarity: Ease of comprehension and interpretability of
|
||||
> structured knowledge.*
|
||||
|
||||
Each task: a free-text input + the expected claim-lattice
|
||||
structure (N claims, ordered, with pointer IDs). Metric: structural-
|
||||
match-rate (sorted-claim-text + pointer-ID-set agreement, NOT
|
||||
exact-string).
|
||||
|
||||
Maps to arborist surfaces: `arborist.qa.parse_claims` +
|
||||
`arborist.pi_star.claim_lattice`.
|
||||
|
||||
### 2.5 Feedback Loop
|
||||
|
||||
> *Integration Efficiency: Speed and efficacy in assimilating
|
||||
> feedback into system processes.
|
||||
> Continuous Improvement: Measurable performance enhancements
|
||||
> through iterative feedback mechanisms.*
|
||||
|
||||
Each task: a sequence of (operation, observation) pairs forming a
|
||||
feedback chain; the system should integrate observations into
|
||||
subsequent operations. Metric: integration-coverage-rate (fraction
|
||||
of observations that affect a downstream snapshot).
|
||||
|
||||
Maps to arborist surfaces: `memory_branch_summaries` updates +
|
||||
`audit_events` chain delta between snapshots.
|
||||
|
||||
---
|
||||
|
||||
## 3. Design choices
|
||||
|
||||
### 3.1 Battery vs sibling layer
|
||||
|
||||
**A. Treat 5F as a peer to 5S/5T inside `bench/batteries/`.** Same
|
||||
runner skeleton; new file `b_5f.py`.
|
||||
|
||||
**B. Treat 5F as a separate top-level harness.** New module
|
||||
`bench/quality/`. Distinct because Function/Finetuning are
|
||||
operational rather than substrate-evaluation properties.
|
||||
|
||||
**Recommendation:** A. Keeps the harness uniform; the runner CLI
|
||||
already supports arbitrary battery names. The semantic distinction
|
||||
between 5S/5T (substrate) and 5F (operational) is documented in
|
||||
the ticket but doesn't need a code-level distinction.
|
||||
|
||||
### 3.2 Fixture sourcing per sub-battery
|
||||
|
||||
| Sub-battery | Fixture source |
|
||||
|---|---|
|
||||
| Function | Hand-curated `(directive, expected-shape, evaluator)` triples. |
|
||||
| Finetuning | Generated from `selfmodel_records` chains in actual shards (or test fixtures). |
|
||||
| Falsification | Planted-error fixtures: known-bad providence records with stated falsification reason. |
|
||||
| Formulate | Hand-curated `(text, expected-claim-lattice)` pairs from existing claim-lattice tests. |
|
||||
| Feedback Loop | Multi-snapshot fixture chains; each chain tagged with expected memory-root delta semantics. |
|
||||
|
||||
### 3.3 Phase staging
|
||||
|
||||
Like 5S/5T, 5F lands in two phases:
|
||||
|
||||
- **Phase 1a (this ticket scope):** runner skeleton in `b_5f.py`,
|
||||
~10 seed fixtures per sub-battery, all sub-batteries return real
|
||||
results (no stubs). Get the surface up.
|
||||
- **Phase 1b (follow-up):** expand fixtures to 30-50 per sub-battery
|
||||
for bench-floor signal.
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation sketch
|
||||
|
||||
### 4.1 Module + fixtures
|
||||
|
||||
```
|
||||
bench/batteries/b_5f.py # new; Function/Finetuning/Falsification/Formulate/Feedback runners
|
||||
bench/fixtures/5f/function-v1.jsonl
|
||||
bench/fixtures/5f/finetuning-v1.jsonl
|
||||
bench/fixtures/5f/falsification-v1.jsonl
|
||||
bench/fixtures/5f/formulate-v1.jsonl
|
||||
bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||
```
|
||||
|
||||
### 4.2 Runner registration
|
||||
|
||||
In `bench/batteries/runner.py`:
|
||||
|
||||
```python
|
||||
_BATTERIES = {
|
||||
"5s": b_5s.SUB_BATTERIES,
|
||||
"5t": b_5t.SUB_BATTERIES,
|
||||
"5f": b_5f.SUB_BATTERIES, # new
|
||||
}
|
||||
```
|
||||
|
||||
Existing `--all` mode picks up the new fixtures automatically once
|
||||
they're in `_DEFAULT_FIXTURES`.
|
||||
|
||||
### 4.3 Makefile additions
|
||||
|
||||
```makefile
|
||||
bench-5f: bootstrap ## 5F battery (Function/Finetuning/Falsification/Formulate/Feedback Loop)
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub function --fixtures bench/fixtures/5f/function-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub finetuning --fixtures bench/fixtures/5f/finetuning-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub falsification --fixtures bench/fixtures/5f/falsification-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub formulate --fixtures bench/fixtures/5f/formulate-v1.jsonl
|
||||
$(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop --fixtures bench/fixtures/5f/feedback-loop-v1.jsonl
|
||||
|
||||
bench-5s5t5f: bench-5s bench-5t bench-5f ## complete Dav1DPrometheus suite
|
||||
```
|
||||
|
||||
### 4.4 Tests
|
||||
|
||||
- `tests/test_bench_batteries.py` extension: positive coverage
|
||||
for each new sub-battery.
|
||||
- Determinism tests: each runner produces identical
|
||||
`BatteryResult.fixture_digest` and `runtime_digest` across
|
||||
back-to-back runs.
|
||||
|
||||
### 4.5 Bumps
|
||||
|
||||
- New module `bench/batteries/b_5f.py`.
|
||||
- 5 new fixture files.
|
||||
- New Makefile targets.
|
||||
- No schema changes; no policy hash changes.
|
||||
|
||||
---
|
||||
|
||||
## 5. Out of scope
|
||||
|
||||
- Phase 1b fixture expansion (target 30-50 per sub-battery). Phase
|
||||
1a's seed set lands first.
|
||||
- Cross-battery composition (e.g., "score this agent across
|
||||
5S+5T+5F"). Aggregation is its own concern.
|
||||
- Calibration of metric thresholds. Phase 1a measures; thresholds
|
||||
for v8 selection acceptance come later.
|
||||
- Auto-grading via LLM. All 5F evaluators are deterministic
|
||||
predicates; LLM-as-judge is future work.
|
||||
|
||||
---
|
||||
|
||||
## 6. Risks & open questions
|
||||
|
||||
- **Function evaluator design.** "Did the system execute the
|
||||
designated task" requires a per-task predicate. v1 ships a
|
||||
small set of canonical evaluator types (shape-match,
|
||||
pointer-set-match, threshold-on-metric). New evaluator types
|
||||
land in follow-ups.
|
||||
- **Finetuning needs SelfModel chains.** Realistic fixtures
|
||||
require real shard data with parent→child SelfModel transitions.
|
||||
Phase 1a uses synthetic chains generated in-test; Phase 1b
|
||||
pulls from production shards.
|
||||
- **Falsification fixtures are fragile.** A planted-error fixture
|
||||
bound to a specific verifier signature may stop catching the
|
||||
error if the verifier changes shape. Mitigation: tag each
|
||||
Falsification fixture with the `verifier_method_root` it was
|
||||
authored against; bench reports flag mismatches.
|
||||
|
||||
---
|
||||
|
||||
## 7. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Larger surface than #000023 / #000024
|
||||
because 5F has five sub-batteries vs three remaining in 5S and
|
||||
four in 5T. Recommended sequencing: ship Phase 1a in one push
|
||||
(seed-fixture per sub-battery, ~10 each), then Phase 1b expands.
|
||||
|
||||
Closure criterion (Phase 1a): `bench/batteries/b_5f.py` lands
|
||||
with non-stub runners for all five sub-batteries; seed fixtures
|
||||
+ Makefile targets pass `make bench-5f` end-to-end. Phase 1b
|
||||
keeps the ticket open in-progress until fixtures are expanded.
|
||||
Loading…
Add table
Add a link
Reference in a new issue