engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench/fixtures/5f/function-v1.jsonl
russell@unturf.com 708aa450cb
fan-out: warrant ladder wiring · witness follow-ups · 5F Phase 1d 
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.

#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================

Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:

- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
  the cached row's question + answer + per-source raw chunks +
  URIs + titles; result lands as `authorship` field alongside the
  other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
  <readable-tier>` when result['authorship'] is populated with a
  non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
  NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
  without an `authorship` key render unchanged.

Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).

#000028 follow-ups — capital ledger + sample-rate
==================================================

Two policy fields layered on top of canonical_witness_enabled:

- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
  wanting passive calibration set 0.05 to fire witness on 5% of
  canonical questions while paying 5% of LLM cost. 0.0 effectively
  off; 1.0 = current always-on behavior. Gating uses random.random()
  so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
  ones). op_type='canonical_witness'; estimator inputs include
  prompt_chars + answer_chars + llm_seconds + agreement_label +
  pi_star_ref. Best-effort: ledger-write failure must never fail
  the query (sidecar discipline).

Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.

Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.

#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================

Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).

  function       — claim_count cycles 2..7 across new fixtures
  falsification  — 10-violation palette across new ids
  feedback-loop  — fact-N learning chains
  finetuning     — capability transitions across canonical π*
                   (math/logic/algebra/calculus pool)
  formulate      — multi-pointer claim shapes

500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).

Tests
=====

Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
2026-05-09 12:42:56 -04:00

51 lines
25 KiB
JSON
Raw Permalink Blame History

{"_meta": {"battery": "5f", "sub_battery": "function", "version": "v1", "task_count": 50, "notes": "Phase 1a: ten seed tasks across shape_match/pointer_set_match/threshold_on_metric evaluators. Phase 1b expands to 30+. Phase 1c (2026-05-09): expanded 10→30. Phase 1d (2026-05-09): expanded 30→50."}}
{"id":"5f-fn-001","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Return exactly two claim-lattice claims with evidence pointers.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"first claim","pointer_ids":["E1"]},{"claim_text":"second claim","pointer_ids":["E2"]}]},"expected":"pass"}
{"id":"5f-fn-002","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Return exactly three claims with pointers.","evaluator":"shape_match","expected_shape":{"claim_count":3,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":["E1"]},{"claim_text":"b","pointer_ids":["E2"]},{"claim_text":"c","pointer_ids":["E3"]}]},"expected":"pass"}
{"id":"5f-fn-003","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect missing pointers in claims.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":[]},{"claim_text":"b","pointer_ids":["E2"]}]},"expected":"fail"}
{"id":"5f-fn-004","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect wrong claim count.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":["E1"]}]},"expected":"fail"}
{"id":"5f-fn-005","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify pointer set matches.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3"]},"produced_output":{"claims":[{"claim_text":"x","pointer_ids":["E1","E2"]},{"claim_text":"y","pointer_ids":["E3"]}]},"expected":"pass"}
{"id":"5f-fn-006","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect pointer-set mismatch.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"produced_output":{"claims":[{"claim_text":"x","pointer_ids":["E1","E5"]}]},"expected":"fail"}
{"id":"5f-fn-007","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify directive coverage threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"directive_coverage","threshold":0.8},"produced_output":{"directive_coverage":0.95},"expected":"pass"}
{"id":"5f-fn-008","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify strict-rate threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"strict_rate","threshold":0.5},"produced_output":{"strict_rate":0.62},"expected":"pass"}
{"id":"5f-fn-009","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect failed metric threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"strict_rate","threshold":0.5},"produced_output":{"strict_rate":0.31},"expected":"fail"}
{"id":"5f-fn-010","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify pointer set with single pointer.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1"]},"produced_output":{"claims":[{"claim_text":"single","pointer_ids":["E1"]}]},"expected":"pass"}
{"id": "5f-fn-011", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 1 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 1, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}]}, "expected": "pass"}
{"id": "5f-fn-012", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
{"id": "5f-fn-013", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 5 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}]}, "expected": "pass"}
{"id": "5f-fn-014", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 6 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}]}, "expected": "pass"}
{"id": "5f-fn-015", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 7 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 7, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}]}, "expected": "pass"}
{"id": "5f-fn-016", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 8 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 8, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}, {"claim_text": "claim 8", "pointer_ids": ["E8"]}]}, "expected": "pass"}
{"id": "5f-fn-017", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (1 vs expected 2).", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}]}, "expected": "fail"}
{"id": "5f-fn-018", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (2 vs expected 3).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}, {"claim_text": "x", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-019", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (3 vs expected 5).", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}, {"claim_text": "x", "pointer_ids": ["E2"]}, {"claim_text": "x", "pointer_ids": ["E3"]}]}, "expected": "fail"}
{"id": "5f-fn-020", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3", "E4"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1", "E2"]}, {"claim_text": "c2", "pointer_ids": ["E3", "E4"]}]}, "expected": "pass"}
{"id": "5f-fn-021", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1", "E2"]}]}, "expected": "pass"}
{"id": "5f-fn-022", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-023", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1"]}, {"claim_text": "c2", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-024", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.5}, "produced_output": {"strict_rate": 0.7}, "expected": "pass"}
{"id": "5f-fn-025", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.5}, "produced_output": {"strict_rate": 0.45}, "expected": "fail"}
{"id": "5f-fn-026", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on directive_coverage.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "directive_coverage", "threshold": 0.9}, "produced_output": {"directive_coverage": 0.99}, "expected": "pass"}
{"id": "5f-fn-027", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on directive_coverage.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "directive_coverage", "threshold": 0.95}, "produced_output": {"directive_coverage": 0.92}, "expected": "fail"}
{"id": "5f-fn-028", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on hybrid_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "hybrid_rate", "threshold": 0.3}, "produced_output": {"hybrid_rate": 0.32}, "expected": "pass"}
{"id": "5f-fn-029", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate (sub-floor).", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.55}, "produced_output": {"strict_rate": 0.4}, "expected": "fail"}
{"id": "5f-fn-030", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.6}, "produced_output": {"strict_rate": 0.6}, "expected": "pass"}
{"id": "5f-fn-031", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 3 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}]}, "expected": "pass"}
{"id": "5f-fn-032", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
{"id": "5f-fn-033", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 5 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}]}, "expected": "pass"}
{"id": "5f-fn-034", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 6 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}]}, "expected": "pass"}
{"id": "5f-fn-035", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 7 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 7, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}]}, "expected": "pass"}
{"id": "5f-fn-036", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 2 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}]}, "expected": "pass"}
{"id": "5f-fn-037", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 3 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}]}, "expected": "pass"}
{"id": "5f-fn-038", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
{"id": "5f-fn-039", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 5 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}]}, "expected": "pass"}
{"id": "5f-fn-040", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 6 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}]}, "expected": "pass"}
{"id": "5f-fn-041", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 7 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 7, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}]}, "expected": "pass"}
{"id": "5f-fn-042", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 2 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}]}, "expected": "pass"}
{"id": "5f-fn-043", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 3 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}]}, "expected": "pass"}
{"id": "5f-fn-044", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
{"id": "5f-fn-045", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 5 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}]}, "expected": "pass"}
{"id": "5f-fn-046", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 6 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}]}, "expected": "pass"}
{"id": "5f-fn-047", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 7 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 7, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}]}, "expected": "pass"}
{"id": "5f-fn-048", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 2 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}]}, "expected": "pass"}
{"id": "5f-fn-049", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 3 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}]}, "expected": "pass"}
{"id": "5f-fn-050", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers (Phase 1d).", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
Reference in a new issue View git blame Copy permalink