engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench/fixtures/5f/function-live-v1.jsonl
russell@unturf.com 708aa450cb
fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.

#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================

Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:

- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
  the cached row's question + answer + per-source raw chunks +
  URIs + titles; result lands as `authorship` field alongside the
  other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
  <readable-tier>` when result['authorship'] is populated with a
  non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
  NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
  without an `authorship` key render unchanged.

Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).

#000028 follow-ups — capital ledger + sample-rate
==================================================

Two policy fields layered on top of canonical_witness_enabled:

- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
  wanting passive calibration set 0.05 to fire witness on 5% of
  canonical questions while paying 5% of LLM cost. 0.0 effectively
  off; 1.0 = current always-on behavior. Gating uses random.random()
  so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
  ones). op_type='canonical_witness'; estimator inputs include
  prompt_chars + answer_chars + llm_seconds + agreement_label +
  pi_star_ref. Best-effort: ledger-write failure must never fail
  the query (sidecar discipline).

Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.

Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.

#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================

Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).

  function       — claim_count cycles 2..7 across new fixtures
  falsification  — 10-violation palette across new ids
  feedback-loop  — fact-N learning chains
  finetuning     — capability transitions across canonical π*
                   (math/logic/algebra/calculus pool)
  formulate      — multi-pointer claim shapes

500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).

Tests
=====

Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
2026-05-09 12:42:56 -04:00

51 lines
17 KiB
JSON
Raw Permalink Blame History

{"_meta": {"battery": "5f", "sub_battery": "function", "version": "v1", "task_count": 50, "notes": "Phase 1b.2: input_text routes through arborist.qa.parse_claims; same shape evaluators used as Phase 1a but on live-parsed output. Phase 1d (2026-05-09): expanded 30→50."}}
{"id":"5f-fn-live-001","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- One claim. [E1]\n- Two claim. [E2]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"pass"}
{"id":"5f-fn-live-002","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Single claim. [E1]","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"pass"}
{"id":"5f-fn-live-003","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Three. [E1]\n- Three. [E2]\n- Three. [E3]","evaluator":"shape_match","expected_shape":{"claim_count":3,"pointers_required":true},"expected":"pass"}
{"id":"5f-fn-live-004","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Pointers expected. [E1]\n- More. [E2]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"expected":"pass"}
{"id":"5f-fn-live-005","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Detect missing pointer. [E1, E2, E3]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3"]},"expected":"pass"}
{"id":"5f-fn-live-006","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"Plain text without pointer.","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"fail"}
{"id":"5f-fn-live-007","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Wrong count. [E1]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"fail"}
{"id":"5f-fn-live-008","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Mismatched. [E5]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1"]},"expected":"fail"}
{"id":"5f-fn-live-009","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Multi-pointer claim. [E1, E2, E3]","evaluator":"shape_match","expected_shape":{"claim_count":1,"pointers_required":true},"expected":"pass"}
{"id":"5f-fn-live-010","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- A long claim with several content tokens. [E1]\n- Another long claim with content. [E2]","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"expected":"pass"}
{"id":"5f-fn-live-011","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- All four named. [E1]\n- All four named. [E2]\n- All four named. [E3]\n- All four named. [E4]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3","E4"]},"expected":"pass"}
{"id":"5f-fn-live-012","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","input_text":"- Subtle mismatch. [E1, E2]\n- Other one. [E3]","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"expected":"fail"}
{"id": "5f-fn-live-013", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Single. [E1]", "evaluator": "shape_match", "expected_shape": {"claim_count": 1, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-014", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- A. [E1]\n- B. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-015", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- A. [E1]\n- B. [E2]\n- C. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-016", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Four. [E1]\n- Four. [E2]\n- Four. [E3]\n- Four. [E4]", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-017", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Five. [E1]\n- Five. [E2]\n- Five. [E3]\n- Five. [E4]\n- Five. [E5]", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-018", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Single. [E5]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E5"]}, "expected": "pass"}
{"id": "5f-fn-live-019", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Pair. [E1, E2]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2"]}, "expected": "pass"}
{"id": "5f-fn-live-020", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- A. [E10]\n- B. [E20]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E10", "E20"]}, "expected": "pass"}
{"id": "5f-fn-live-021", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Wrong count for shape_match. [E1]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "fail"}
{"id": "5f-fn-live-022", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Pointer set differs. [E5]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1"]}, "expected": "fail"}
{"id": "5f-fn-live-023", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- One. [E1]\n- Two. [E2]\n- Three. [E3]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3"]}, "expected": "pass"}
{"id": "5f-fn-live-024", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Heavy multi pointer. [E1, E2, E3, E4, E5]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3", "E4", "E5"]}, "expected": "pass"}
{"id": "5f-fn-live-025", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Strict. [E1]\n- Hybrid. [E2]\n- Ungrounded. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-026", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Reasonable claim with content. [E1]", "evaluator": "shape_match", "expected_shape": {"claim_count": 1, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-027", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- More. [E11]\n- More. [E12]\n- More. [E13]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E11", "E12", "E13"]}, "expected": "pass"}
{"id": "5f-fn-live-028", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Mismatch test. [E1]\n- Other. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "fail"}
{"id": "5f-fn-live-029", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Reject this shape. [E1, E2]", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1"]}, "expected": "fail"}
{"id": "5f-fn-live-030", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Edge case. [E99]", "evaluator": "shape_match", "expected_shape": {"claim_count": 1, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-031", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-032", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-033", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-034", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]\n- Claim 6. [E6]", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-035", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-036", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-037", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-038", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-039", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]\n- Claim 6. [E6]", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-040", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-041", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-042", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-043", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-044", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]\n- Claim 6. [E6]", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-045", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-046", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-047", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-048", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-049", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]\n- Claim 3. [E3]\n- Claim 4. [E4]\n- Claim 5. [E5]\n- Claim 6. [E6]", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "expected": "pass"}
{"id": "5f-fn-live-050", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "input_text": "- Claim 1. [E1]\n- Claim 2. [E2]", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "expected": "pass"}
Reference in a new issue View git blame Copy permalink