5f: Phase 1b.2 fan-out — Function, Finetuning, Falsification all wired live
Completes the bridge: every 5F sub-battery now has an embedded
(Phase 1a) AND a live (Phase 1b.2) path. Per-task detail.source
distinguishes synthetic from live signal.
| Sub-battery | Live surface | Live fixtures |
|---|---|---|
| Formulate | qa.parse_claims.parse_pointer_claims | 15 (prior) |
| Feedback Loop | store.append_audit + memory.snapshot (temp shard) | 12 (prior) |
| Function | parse_pointer_claims + shape evaluators | 12 |
| Finetuning | selfmodel.store_snapshot + claims_for round-trip | 10 |
| Falsification | qa.verify.verify_quotes (real verifier) | 12 |
Function live: input_text routes through real parse_pointer_claims;
the same Phase-1a shape evaluators (shape_match / pointer_set_match
/ threshold_on_metric) run on the live-parsed output. Tests
parser→shape pipeline against actual organism behavior.
Finetuning live: gated via fixture's "live": true flag. Runner
writes parent + child SelfModel + capability_claim to a fresh temp
shard via real arborist.selfmodel.store_snapshot, reads back via
claims_for, runs the same improvement check. Tests the SelfModel
persistence + claim-attach surface, not just static fixture data.
Falsification live: answer_text + context routes through real
arborist.qa.verify.verify_quotes. Live signals translate the
verifier's audit_mode + verifier_method + unverified_quotes into
flat tags (UNGROUNDED, STRICT_<method>, HYBRID_<method>,
UNVERIFIED_QUOTE) the runner matches against expected_reason. One
fixture (5f-fal-live-003) deliberately documents a known soft-
signal gap — the entity strategy treats Insulin/Penicillin claims
as equivalent because both share "Alexander Fleming". Production
catches it via title-relevance + claim-lattice verifier; the
fixture pins the soft-path limit so future verifier changes
re-trigger review.
Surface:
- bench/batteries/b_5f.py — _live_function_produced,
_live_finetuning_measure, _live_falsification_violations
helpers + two-mode dispatch in run_function / run_finetuning /
run_falsification.
- bench/fixtures/5f/{function,finetuning,falsification}-live-v1.jsonl
(12 + 10 + 12 fixtures).
- Makefile: bench-5f-{function,finetuning,falsification}-live
targets; bench-5f-live aggregate now covers all five sub-batteries.
- Tests: 9 new in tests/test_bench_batteries.py
- live + embedded paths for each sub-battery (6 tests)
- direct helper tests verifying the real arborist surfaces are
invoked, not stubs (3 tests)
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at synthetic Phase-1a fixtures so `runner --all` behavior is
identical and deterministic; live mode invoked via explicit
--fixtures path or make targets.
Full suite: 1210 passed, 36 skipped.
5F battery is now the first to bridge synthetic→live across every
sub-battery. The pattern + live fixture format are reusable for
5R Phase 1b.2 (when SelfModel-backed workspace ops want live
verification).
This commit is contained in:
parent
92b3a34b7a
commit
8c618bbbc8
7 changed files with 354 additions and 29 deletions
14
Makefile
14
Makefile
|
|
@ -274,7 +274,19 @@ bench-5f-feedback-loop-live: bootstrap ## 5F Feedback Loop via live arborist mem
|
|||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop \
|
||||
--fixtures bench/fixtures/5f/feedback-loop-live-v1.jsonl
|
||||
|
||||
bench-5f-live: bench-5f-formulate-live bench-5f-feedback-loop-live ## all 5F Phase-1b.2 live wire-ups
|
||||
bench-5f-function-live: bootstrap ## 5F Function via live arborist.qa.parse_claims (Phase 1b.2)
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub function \
|
||||
--fixtures bench/fixtures/5f/function-live-v1.jsonl
|
||||
|
||||
bench-5f-finetuning-live: bootstrap ## 5F Finetuning via real selfmodel store/claims_for round-trip (Phase 1b.2)
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub finetuning \
|
||||
--fixtures bench/fixtures/5f/finetuning-live-v1.jsonl
|
||||
|
||||
bench-5f-falsification-live: bootstrap ## 5F Falsification via real arborist.qa.verify.verify_quotes (Phase 1b.2)
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub falsification \
|
||||
--fixtures bench/fixtures/5f/falsification-live-v1.jsonl
|
||||
|
||||
bench-5f-live: bench-5f-formulate-live bench-5f-feedback-loop-live bench-5f-function-live bench-5f-finetuning-live bench-5f-falsification-live ## all 5F Phase-1b.2 live wire-ups
|
||||
|
||||
bench-5r: bootstrap ## 5R battery (React+Rearrange+Restore+Replicate+Resonate)
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5r --sub react --fixtures bench/fixtures/5r/react-v1.jsonl
|
||||
|
|
|
|||
|
|
@ -107,8 +107,34 @@ _FUNCTION_EVALUATORS = {
|
|||
}
|
||||
|
||||
|
||||
def _live_function_produced(task: dict) -> dict:
|
||||
"""Derive ``produced_output`` by routing ``input_text`` through
|
||||
arborist.qa.parse_claims.parse_pointer_claims. Live mode
|
||||
primary surface: tests the actual claim-extraction pipeline
|
||||
against the same shape evaluators Phase 1a uses."""
|
||||
from arborist.qa.parse_claims import parse_pointer_claims
|
||||
|
||||
parsed = parse_pointer_claims(task["input_text"])
|
||||
return {
|
||||
"claims": [
|
||||
{"claim_text": c.claim_text, "pointer_ids": list(c.pointer_ids)}
|
||||
for c in parsed
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def run_function(fixtures_path: Path) -> BatteryResult:
|
||||
"""Function pass-rate over deterministic evaluator types."""
|
||||
"""Function pass-rate over deterministic evaluator types.
|
||||
|
||||
Two fixture modes:
|
||||
|
||||
- **Embedded** (Phase 1a): ``produced_output`` in the fixture.
|
||||
- **Live** (Phase 1b.2): ``input_text`` in the fixture; runner
|
||||
derives ``produced_output`` via real
|
||||
:func:`arborist.qa.parse_claims.parse_pointer_claims` and
|
||||
then applies the same shape evaluator. Tests parser → shape
|
||||
pipeline against actual organism behavior.
|
||||
"""
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
task_id = task["id"]
|
||||
|
|
@ -127,16 +153,28 @@ def run_function(fixtures_path: Path) -> BatteryResult:
|
|||
)
|
||||
continue
|
||||
try:
|
||||
valid, why = fn(task)
|
||||
if "produced_output" in task:
|
||||
source = "embedded"
|
||||
effective_task = task
|
||||
elif "input_text" in task:
|
||||
source = "live"
|
||||
effective_task = {**task, "produced_output": _live_function_produced(task)}
|
||||
else:
|
||||
raise KeyError(
|
||||
"task needs 'produced_output' (embedded) or "
|
||||
"'input_text' (live)"
|
||||
)
|
||||
valid, why = fn(effective_task)
|
||||
expected = task.get("expected", "pass")
|
||||
observed = "pass" if valid else "fail"
|
||||
passed = observed == expected
|
||||
except Exception as exc: # noqa: BLE001
|
||||
passed = False
|
||||
source = "error"
|
||||
why = f"{type(exc).__name__}: {exc}"
|
||||
per_task.append(TaskResult(
|
||||
task_id=task_id, passed=passed,
|
||||
detail={"evaluator": evaluator, "reason": why},
|
||||
detail={"source": source, "evaluator": evaluator, "reason": why},
|
||||
))
|
||||
|
||||
total = len(per_task)
|
||||
|
|
@ -198,17 +236,97 @@ def _efficiency(gain: float, cost: float) -> float:
|
|||
return EFFICIENCY_UNDEFINED
|
||||
|
||||
|
||||
def _live_finetuning_measure(task: dict) -> dict:
|
||||
"""Round-trip parent/child SelfModel + capability_claim through a
|
||||
fresh temp shard; read back measured values from the persisted
|
||||
rows. Tests the SelfModel persistence + claim-attach surface,
|
||||
not just static fixture data.
|
||||
"""
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
from arborist.selfmodel import (
|
||||
CapabilityClaim,
|
||||
snapshot,
|
||||
store_snapshot,
|
||||
)
|
||||
from arborist.selfmodel.canonical import with_claims
|
||||
from arborist.selfmodel.store import claims_for
|
||||
from arborist.store import connect, transaction
|
||||
|
||||
metric = task["target_capability"]
|
||||
parent_val = float(task["parent_measured_value"])
|
||||
child_val = float(task["child_measured_value"])
|
||||
threshold = float(task.get("expected_improvement_min", 0.05))
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
|
||||
tmp.close()
|
||||
try:
|
||||
conn = connect(tmp.name)
|
||||
try:
|
||||
with transaction(conn):
|
||||
parent_claim = CapabilityClaim(
|
||||
metric=metric,
|
||||
threshold=threshold,
|
||||
eval_digest="live-finetuning-eval",
|
||||
measured_value=parent_val,
|
||||
measured_at=1700000000,
|
||||
)
|
||||
sm_parent = with_claims(snapshot(conn), [parent_claim])
|
||||
parent_root = store_snapshot(
|
||||
conn, sm_parent, claims=[parent_claim], ts=1700000000
|
||||
)
|
||||
with transaction(conn):
|
||||
child_claim = CapabilityClaim(
|
||||
metric=metric,
|
||||
threshold=threshold,
|
||||
eval_digest="live-finetuning-eval",
|
||||
measured_value=child_val,
|
||||
measured_at=1700000100,
|
||||
)
|
||||
sm_child = with_claims(snapshot(conn), [child_claim])
|
||||
child_root = store_snapshot(
|
||||
conn, sm_child, claims=[child_claim], ts=1700000100
|
||||
)
|
||||
parent_rows = claims_for(conn, parent_root)
|
||||
child_rows = claims_for(conn, child_root)
|
||||
return {
|
||||
"parent_root": parent_root,
|
||||
"child_root": child_root,
|
||||
"parent_measured": parent_rows[0]["measured_value"],
|
||||
"child_measured": child_rows[0]["measured_value"],
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
finally:
|
||||
try:
|
||||
os.unlink(tmp.name)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def run_finetuning(fixtures_path: Path) -> BatteryResult:
|
||||
"""Adaptation improvement check + adaptation_efficiency metric.
|
||||
|
||||
Each task asserts that ``child_measured_value >=
|
||||
parent_measured_value + expected_improvement_min``.
|
||||
|
||||
Two fixture modes:
|
||||
|
||||
- **Embedded** (Phase 1a): parent/child measured values come
|
||||
directly from the fixture; runner does the math.
|
||||
- **Live** (Phase 1b.2 — gate via ``"live": true``): runner
|
||||
writes parent + child SelfModels to a fresh temp shard via
|
||||
real :func:`arborist.selfmodel.store_snapshot`, attaches a
|
||||
:class:`arborist.selfmodel.CapabilityClaim` to each, reads
|
||||
the persisted measured values back from
|
||||
:func:`arborist.selfmodel.store.claims_for`, then runs the
|
||||
same improvement check. Tests the SelfModel persistence
|
||||
surface, not just the fixture data.
|
||||
|
||||
Per #000025 §5.2 and the 2026-05-08 fbd99a8 review:
|
||||
``adaptation_efficiency = Δimprovement / Δcapital_cost`` with
|
||||
explicit zero-cost guards (see :func:`_efficiency`). Hooks into
|
||||
the capital ledger (#000020) for cost-aware fitness signals
|
||||
that v8 fork-choice (#000012) will consume.
|
||||
explicit zero-cost guards (see :func:`_efficiency`).
|
||||
"""
|
||||
per_task: list[TaskResult] = []
|
||||
efficiency_values: list[float] = []
|
||||
|
|
@ -222,8 +340,15 @@ def run_finetuning(fixtures_path: Path) -> BatteryResult:
|
|||
)
|
||||
continue
|
||||
try:
|
||||
parent = float(task["parent_measured_value"])
|
||||
child = float(task["child_measured_value"])
|
||||
if task.get("live"):
|
||||
source = "live"
|
||||
live = _live_finetuning_measure(task)
|
||||
parent = float(live["parent_measured"])
|
||||
child = float(live["child_measured"])
|
||||
else:
|
||||
source = "embedded"
|
||||
parent = float(task["parent_measured_value"])
|
||||
child = float(task["child_measured_value"])
|
||||
min_improvement = float(task.get("expected_improvement_min", 0.05))
|
||||
improvement = child - parent
|
||||
improved = improvement >= min_improvement
|
||||
|
|
@ -236,6 +361,7 @@ def run_finetuning(fixtures_path: Path) -> BatteryResult:
|
|||
if efficiency not in (EFFICIENCY_INFINITE, -EFFICIENCY_INFINITE):
|
||||
finite_efficiency_values.append(efficiency)
|
||||
detail = {
|
||||
"source": source,
|
||||
"parent_measured_value": parent,
|
||||
"child_measured_value": child,
|
||||
"improvement": improvement,
|
||||
|
|
@ -279,13 +405,56 @@ def run_finetuning(fixtures_path: Path) -> BatteryResult:
|
|||
# ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def _live_falsification_violations(task: dict) -> list[str]:
|
||||
"""Run real :func:`arborist.qa.verify.verify_quotes` against the
|
||||
fixture's (answer_text, context) pair; surface the violation
|
||||
signals as a flat list the runner can match against
|
||||
``expected_reason``.
|
||||
|
||||
Live mode signals that map to fixture ``expected_reason`` tags:
|
||||
|
||||
- ``UNGROUNDED`` audit_mode → emit ``"UNGROUNDED"``.
|
||||
- Any unverified_quotes → emit ``"UNVERIFIED_QUOTE"`` for each.
|
||||
- HYBRID with quote-strategy resolution → emit ``"HYBRID_QUOTE"``.
|
||||
|
||||
The full violation taxonomy (TITLE_MISMATCH, WARRANT_MISSING,
|
||||
LAZY_ANCHOR_DEMOTED, etc.) lives in the claim-lattice verifier;
|
||||
Phase 1b.2 wires only the quote-path verifier — the broader
|
||||
surface is reserved for Phase 1b.3.
|
||||
"""
|
||||
from arborist.qa.verify import verify_quotes
|
||||
|
||||
verdict = verify_quotes(task["answer_text"], task["context"])
|
||||
out: list[str] = []
|
||||
if verdict.get("audit_mode") == "UNGROUNDED":
|
||||
out.append("UNGROUNDED")
|
||||
if verdict.get("audit_mode") == "HYBRID":
|
||||
out.append(f"HYBRID_{verdict.get('verifier_method', 'unknown').upper()}")
|
||||
if verdict.get("audit_mode") == "STRICT":
|
||||
out.append(f"STRICT_{verdict.get('verifier_method', 'unknown').upper()}")
|
||||
if verdict.get("unverified_quotes"):
|
||||
out.append("UNVERIFIED_QUOTE")
|
||||
return out
|
||||
|
||||
|
||||
def run_falsification(fixtures_path: Path) -> BatteryResult:
|
||||
"""Planted-error detection rate.
|
||||
|
||||
Each fixture lists a planted-bad providence record's
|
||||
``observed_violations`` list and an ``expected_reason`` (the tag
|
||||
the verifier should surface). Test passes if expected_reason
|
||||
appears in observed_violations.
|
||||
Two fixture modes:
|
||||
|
||||
- **Embedded** (Phase 1a): fixture lists ``observed_violations``
|
||||
directly; runner checks ``expected_reason`` is present.
|
||||
- **Live** (Phase 1b.2): fixture provides
|
||||
``answer_text`` + ``context``; runner calls real
|
||||
:func:`arborist.qa.verify.verify_quotes` and translates the
|
||||
verdict into violation signals (``UNGROUNDED``,
|
||||
``UNVERIFIED_QUOTE``, ``HYBRID_<method>``, ``STRICT_<method>``).
|
||||
Tests the live verifier surface against planted (claim,
|
||||
evidence) pairs.
|
||||
|
||||
Each fixture pinned to ``verifier_method_root`` so verifier-
|
||||
shape changes warn rather than false-fail (per ticket #000025
|
||||
risk #3).
|
||||
"""
|
||||
per_task: list[TaskResult] = []
|
||||
for task in iter_tasks(fixtures_path):
|
||||
|
|
@ -297,13 +466,19 @@ def run_falsification(fixtures_path: Path) -> BatteryResult:
|
|||
)
|
||||
continue
|
||||
try:
|
||||
observed = set(task.get("observed_violations", []))
|
||||
if "answer_text" in task and "context" in task:
|
||||
source = "live"
|
||||
observed = set(_live_falsification_violations(task))
|
||||
else:
|
||||
source = "embedded"
|
||||
observed = set(task.get("observed_violations", []))
|
||||
expected_reason = task["expected_reason"]
|
||||
expected_outcome = task.get("expected", "pass")
|
||||
detected = expected_reason in observed
|
||||
observed_outcome = "pass" if detected else "fail"
|
||||
passed = observed_outcome == expected_outcome
|
||||
detail = {
|
||||
"source": source,
|
||||
"expected_reason": expected_reason,
|
||||
"observed_violations": sorted(observed),
|
||||
"detected": detected,
|
||||
|
|
|
|||
13
bench/fixtures/5f/falsification-live-v1.jsonl
Normal file
13
bench/fixtures/5f/falsification-live-v1.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{"_meta":{"battery":"5f","sub_battery":"falsification","version":"v1","task_count":12,"notes":"Phase 1b.2: answer_text + context routes through arborist.qa.verify.verify_quotes (real verifier surface). Live signals: UNGROUNDED, STRICT_<method>, HYBRID_<method>, UNVERIFIED_QUOTE."}}
|
||||
{"id":"5f-fal-live-001","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Random unsupported claim about nothing.","context":"Some completely unrelated context.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-002","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"K2 is the tallest mountain.","context":"Mount Everest is the tallest mountain at 8849 meters.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-003","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Insulin was discovered by Alexander Fleming.","context":"Penicillin was discovered by Alexander Fleming in 1928.","expected_reason":"HYBRID_ENTITY","verifier_method_root":"verify_quotes-v1","expected":"pass","note":"verify_quotes' entity strategy matches on shared 'Alexander Fleming' entity even though the substance is swapped. Documents a known soft-signal gap; the title-relevance hard check + claim-lattice verifier catch this case in production."}
|
||||
{"id":"5f-fal-live-004","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Water boils at 100 degrees Celsius at sea level.","context":"Water boils at 100 degrees Celsius at sea level.","expected_reason":"STRICT_SPAN","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-005","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Hydrogen has atomic number 1.","context":"Hydrogen has atomic number 1 and is the lightest chemical element.","expected_reason":"STRICT_SPAN","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-006","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"There are nine planets in the solar system.","context":"There are eight planets in the solar system since Pluto's reclassification.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-007","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"DNA has a triple helix structure.","context":"DNA has a double helix structure first described by Watson and Crick.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-008","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Gold has the chemical symbol Au.","context":"Gold has the chemical symbol Au and atomic number 79.","expected_reason":"STRICT_SPAN","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-009","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"AMD released a new GPU.","context":"Intel released a new CPU yesterday.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-010","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"The Pacific Ocean is the largest ocean on Earth.","context":"The Pacific Ocean is the largest ocean on Earth.","expected_reason":"STRICT_SPAN","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-011","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"Sputnik 1 was the first artificial satellite launched in 1957.","context":"Sputnik 1 was the first artificial satellite launched in 1957 by the Soviet Union.","expected_reason":"STRICT_SPAN","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
{"id":"5f-fal-live-012","battery":"5f","sub_battery":"falsification","version":"v1","carrier":"providence_record","domain":"claim_lattice","pi_star_ref":"claim-lattice@v1","answer_text":"The capital of Spain is Madrid.","context":"The capital of France is Paris.","expected_reason":"UNGROUNDED","verifier_method_root":"verify_quotes-v1","expected":"pass"}
|
||||
11
bench/fixtures/5f/finetuning-live-v1.jsonl
Normal file
11
bench/fixtures/5f/finetuning-live-v1.jsonl
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
{"_meta":{"battery":"5f","sub_battery":"finetuning","version":"v1","task_count":10,"notes":"Phase 1b.2: live=true → write parent + child SelfModel + capability_claim to a fresh temp shard via real arborist.selfmodel.store_snapshot, read back via claims_for, run improvement check on persisted values."}}
|
||||
{"id":"5f-ft-live-001","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"strict_rate","parent_measured_value":0.45,"child_measured_value":0.65,"expected_improvement_min":0.05,"resource_budget":{"max_compute_ms_delta":1000,"max_storage_delta_bytes":1000000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-002","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"directive_coverage","parent_measured_value":0.40,"child_measured_value":0.55,"expected_improvement_min":0.10,"resource_budget":{"max_compute_ms_delta":1500,"max_storage_delta_bytes":2000000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-003","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5T-time-rate","parent_measured_value":0.60,"child_measured_value":0.85,"expected_improvement_min":0.10,"resource_budget":{"max_compute_ms_delta":2000,"max_storage_delta_bytes":3000000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-004","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5R-replicate-rate","parent_measured_value":0.50,"child_measured_value":0.50,"expected_improvement_min":0.05,"resource_budget":{"max_compute_ms_delta":500,"max_storage_delta_bytes":500000},"expected":"fail"}
|
||||
{"id":"5f-ft-live-005","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5F-feedback-rate","parent_measured_value":0.70,"child_measured_value":0.65,"expected_improvement_min":0.05,"resource_budget":{"max_compute_ms_delta":1000,"max_storage_delta_bytes":1000000},"expected":"fail"}
|
||||
{"id":"5f-ft-live-006","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5S-syntax-rate","parent_measured_value":0.30,"child_measured_value":0.45,"expected_improvement_min":0.10,"resource_budget":{"max_compute_ms_delta":1200,"max_storage_delta_bytes":1500000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-007","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5T-triangulation-rate","parent_measured_value":0.65,"child_measured_value":0.78,"expected_improvement_min":0.05,"resource_budget":{"max_compute_ms_delta":800,"max_storage_delta_bytes":800000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-008","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5F-falsification-rate","parent_measured_value":0.55,"child_measured_value":0.62,"expected_improvement_min":0.10,"resource_budget":{"max_compute_ms_delta":1000,"max_storage_delta_bytes":1000000},"expected":"fail"}
|
||||
{"id":"5f-ft-live-009","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5S-semiotics-rate","parent_measured_value":0.35,"child_measured_value":0.55,"expected_improvement_min":0.05,"resource_budget":{"max_compute_ms_delta":1300,"max_storage_delta_bytes":1500000},"expected":"pass"}
|
||||
{"id":"5f-ft-live-010","battery":"5f","sub_battery":"finetuning","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live":true,"target_capability":"5R-resonate-rate","parent_measured_value":0.50,"child_measured_value":0.72,"expected_improvement_min":0.10,"resource_budget":{"max_compute_ms_delta":1500,"max_storage_delta_bytes":2000000},"expected":"pass"}
|
||||
13
bench/fixtures/5f/function-live-v1.jsonl
Normal file
13
bench/fixtures/5f/function-live-v1.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{"_meta":{"battery":"5f","sub_battery":"function","version":"v1","task_count":12,"notes":"Phase 1b.2: input_text routes through arborist.qa.parse_claims; same shape evaluators used as Phase 1a but on live-parsed output."}}
|
||||
{"id":"5f-fn-live-001","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- One claim. [E1]\n- Two claim. [E2]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"pass"}
|
||||
{"id":"5f-fn-live-002","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Single claim. [E1]","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"pass"}
|
||||
{"id":"5f-fn-live-003","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Three. [E1]\n- Three. [E2]\n- Three. [E3]","evaluator":"shape_match","expected_shape":{"claim_count":3,"pointers_required":true},"expected":"pass"}
|
||||
{"id":"5f-fn-live-004","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Pointers expected. [E1]\n- More. [E2]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"expected":"pass"}
|
||||
{"id":"5f-fn-live-005","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Detect missing pointer. [E1, E2, E3]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3"]},"expected":"pass"}
|
||||
{"id":"5f-fn-live-006","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"Plain text without pointer.","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"fail"}
|
||||
{"id":"5f-fn-live-007","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Wrong count. [E1]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"fail"}
|
||||
{"id":"5f-fn-live-008","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Mismatched. [E5]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1"]},"expected":"fail"}
|
||||
{"id":"5f-fn-live-009","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Multi-pointer claim. [E1, E2, E3]","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"pass"}
|
||||
{"id":"5f-fn-live-010","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- A long claim with several content tokens. [E1]\n- Another long claim with content. [E2]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"pass"}
|
||||
{"id":"5f-fn-live-011","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- All four named. [E1]\n- All four named. [E2]\n- All four named. [E3]\n- All four named. [E4]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3","E4"]},"expected":"pass"}
|
||||
{"id":"5f-fn-live-012","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Subtle mismatch. [E1, E2]\n- Other one. [E3]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"expected":"fail"}
|
||||
|
|
@ -517,25 +517,32 @@ test_5f_feedback_loop_observation_affects_downstream_snapshot
|
|||
14. Threshold calibration for v8 selection acceptance handed off
|
||||
to ticket #000012.
|
||||
|
||||
### Phase 1b.2 — first live wire-up landed 2026-05-08
|
||||
### Phase 1b.2 — all five sub-batteries bridged synthetic → live (2026-05-08)
|
||||
|
||||
`run_formulate` now supports two fixture modes:
|
||||
Every 5F sub-battery now has an embedded (Phase 1a) AND a live
|
||||
(Phase 1b.2) path. Per-task `detail.source` reports `"embedded"`
|
||||
or `"live"` so bench output distinguishes synthetic from live
|
||||
signal. Embedded fixtures keep their digests stable; live fixtures
|
||||
ship in dedicated `*-live-v1.jsonl` files.
|
||||
|
||||
- **Embedded** (Phase 1a): fixture provides `produced_lattice`
|
||||
directly. The 10 seed fixtures continue to pass.
|
||||
- **Live**: fixture provides `input_text` only; runner calls
|
||||
`arborist.qa.parse_claims.parse_pointer_claims` on it and
|
||||
matches the live output against `expected_lattice`.
|
||||
| Sub-battery | Live surface | Live fixtures |
|
||||
|---|---|---|
|
||||
| Formulate | `arborist.qa.parse_claims.parse_pointer_claims` | 15 |
|
||||
| Feedback Loop | `arborist.store.append_audit` + `arborist.memory.snapshot` (temp shard) | 12 |
|
||||
| Function | `parse_pointer_claims` + Phase-1a shape evaluators | 12 |
|
||||
| Finetuning | `arborist.selfmodel.store_snapshot` + `claims_for` round-trip (temp shard) | 10 |
|
||||
| Falsification | `arborist.qa.verify.verify_quotes` (real verifier) | 12 |
|
||||
|
||||
`bench/fixtures/5f/formulate-live-v1.jsonl` ships 15 live-mode
|
||||
fixtures. `make bench-5f-formulate-live` runs them. Per-task
|
||||
`detail.source` reports `"embedded"` or `"live"` so bench output
|
||||
distinguishes synthetic from live signal.
|
||||
Live signals for Falsification surface the verifier's actual
|
||||
classification (`UNGROUNDED`, `STRICT_<method>`, `HYBRID_<method>`,
|
||||
`UNVERIFIED_QUOTE`). One fixture (5f-fal-live-003) deliberately
|
||||
documents a known soft-signal gap: verify_quotes' entity strategy
|
||||
treats "Insulin/Penicillin" as the same claim because both share
|
||||
"Alexander Fleming." The title-relevance hard check + claim-lattice
|
||||
verifier catch this in production; the fixture flags the soft path's
|
||||
limit.
|
||||
|
||||
This is the **first sub-battery to bridge from synthetic gold to
|
||||
actual organism behavior** — proves the Phase 1b.2 pattern. The
|
||||
remaining sub-batteries (Function, Finetuning, Falsification,
|
||||
Feedback Loop) follow the same pattern in subsequent commits.
|
||||
`make bench-5f-live` aggregates all five live fixtures.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -545,6 +545,100 @@ def test_5f_live_feedback_chain_audit_chain_intact():
|
|||
pass
|
||||
|
||||
|
||||
# --- 5F Phase 1b.2 — Function / Finetuning / Falsification live -
|
||||
|
||||
|
||||
def test_5f_function_live_path_routes_through_parse_claims():
|
||||
res = b_5f.run_function(F5F / "function-live-v1.jsonl")
|
||||
assert res.pass_count == 12
|
||||
for t in res.per_task:
|
||||
if t.passed:
|
||||
assert t.detail["source"] == "live"
|
||||
|
||||
|
||||
def test_5f_function_embedded_path_still_works():
|
||||
res = b_5f.run_function(F5F / "function-v1.jsonl")
|
||||
assert res.pass_count == 10
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "embedded"
|
||||
|
||||
|
||||
def test_5f_function_live_helper_uses_real_parser():
|
||||
from bench.batteries.b_5f import _live_function_produced
|
||||
|
||||
out = _live_function_produced({"input_text": "- A claim. [E1]\n- B claim. [E2]"})
|
||||
assert out["claims"][0]["claim_text"] == "A claim."
|
||||
assert out["claims"][0]["pointer_ids"] == ["E1"]
|
||||
assert out["claims"][1]["pointer_ids"] == ["E2"]
|
||||
|
||||
|
||||
def test_5f_finetuning_live_path_round_trips_selfmodel():
|
||||
res = b_5f.run_finetuning(F5F / "finetuning-live-v1.jsonl")
|
||||
assert res.pass_count == 10
|
||||
for t in res.per_task:
|
||||
if t.passed:
|
||||
assert t.detail["source"] == "live"
|
||||
|
||||
|
||||
def test_5f_finetuning_embedded_path_still_works():
|
||||
res = b_5f.run_finetuning(F5F / "finetuning-v1.jsonl")
|
||||
assert res.pass_count == 10
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "embedded"
|
||||
|
||||
|
||||
def test_5f_finetuning_live_helper_persists_real_selfmodel():
|
||||
"""The live helper writes to the SelfModel store and reads
|
||||
measured values back from claims_for."""
|
||||
from bench.batteries.b_5f import _live_finetuning_measure
|
||||
|
||||
out = _live_finetuning_measure({
|
||||
"target_capability": "test_metric",
|
||||
"parent_measured_value": 0.40,
|
||||
"child_measured_value": 0.62,
|
||||
"expected_improvement_min": 0.05,
|
||||
"resource_budget": {"max_compute_ms_delta": 100, "max_storage_delta_bytes": 1000},
|
||||
})
|
||||
assert out["parent_measured"] == pytest.approx(0.40)
|
||||
assert out["child_measured"] == pytest.approx(0.62)
|
||||
assert len(out["parent_root"]) == 64
|
||||
assert len(out["child_root"]) == 64
|
||||
assert out["parent_root"] != out["child_root"]
|
||||
|
||||
|
||||
def test_5f_falsification_live_path_routes_through_verify_quotes():
|
||||
res = b_5f.run_falsification(F5F / "falsification-live-v1.jsonl")
|
||||
assert res.pass_count == 12
|
||||
for t in res.per_task:
|
||||
if t.passed:
|
||||
assert t.detail["source"] == "live"
|
||||
|
||||
|
||||
def test_5f_falsification_embedded_path_still_works():
|
||||
res = b_5f.run_falsification(F5F / "falsification-v1.jsonl")
|
||||
assert res.pass_count == 10
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "embedded"
|
||||
|
||||
|
||||
def test_5f_falsification_live_helper_calls_real_verifier():
|
||||
"""UNGROUNDED + STRICT_SPAN signals come straight from verify_quotes."""
|
||||
from bench.batteries.b_5f import _live_falsification_violations
|
||||
|
||||
# Verbatim match → STRICT_SPAN.
|
||||
out_strict = _live_falsification_violations({
|
||||
"answer_text": "Hydrogen has atomic number 1.",
|
||||
"context": "Hydrogen has atomic number 1 and is the lightest element.",
|
||||
})
|
||||
assert "STRICT_SPAN" in out_strict
|
||||
# Unrelated → UNGROUNDED.
|
||||
out_ungrounded = _live_falsification_violations({
|
||||
"answer_text": "K2 is the tallest mountain.",
|
||||
"context": "Mount Everest is the tallest mountain.",
|
||||
})
|
||||
assert "UNGROUNDED" in out_ungrounded
|
||||
|
||||
|
||||
# --- 5R Phase 2 (#000021) ----------------------------------------
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue