engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/bench/fixtures/5f/function-v1.jsonl
russell@unturf.com 60b5748ff9
fan-out: ForkScore CLI · authorship warrant ladder · 5F Phase 1c 
Three streams in one commit since they're independent and each is
small.

#000012 Phase 1b — ForkScore CLI surface
========================================

`arborist v8 score` already existed; this adds `--out` for JSON-
artifact emission so CI / downstream graders / mesh peers can
ingest without parsing stdout. New Makefile targets:

- `make bench-fork-baseline` — pins current bench-suite output as
  the ForkScore parent (one-shot per iteration).
- `make bench-fork-score` — runs bench-suite again, scores child
  vs pinned parent, writes bench/results/fork_score_report.json.
  Exit 1 on REJECT so CI can gate.

`FORK_PARENT` / `FORK_CHILD` / `FORK_REPORT` env-vars override
default paths. New regression test pins the --out contract:
stdout and file are byte-identical artifacts; --out auto-creates
parent directories.

#000026 Phase 3 — authorship warrant ladder
============================================

Sidecar classifier in `arborist/qa/warrant_authorship.py`. Six
tiers strongest-to-weakest: AUTHOR_PACKAGE_METADATA →
AUTHOR_REPOSITORY_OWNER → AUTHOR_PAGE_BYLINE →
AUTHOR_PRIMARY_PAGE_TITLE → AUTHOR_COPYRIGHT_FOOTER →
AUTHOR_SECONDARY_SOURCE. Plus NO_AUTHORSHIP_SIGNAL when the
question doesn't smell like authorship (sidecar stays quiet).

Detector regexes for each tier:
- Tier 1: `author = "X"` simple form + TOML inline-table
  `authors = [{ name = "X" }]` form (PEP 621).
- Tier 2: github.com / gitlab.com / codeberg.org / bitbucket.org
  URL pattern.
- Tier 3: "By NAME" / "Author: NAME" prose + <meta name="author">.
  Inline-flag regex keeps the prefix case-insensitive while the
  capitalized-name capture stays case-sensitive.
- Tier 4: cited evidence is the entity's own primary page (host
  tokens overlap title + answer; third-party indexers like
  wikipedia.org explicitly excluded).
- Tier 5: `© NAME` / `Copyright YYYY NAME` (the current
  `virt-back` warrant).
- Tier 6: fall-through when authorship-shaped question hits cited
  evidence with no direct markers.

Sidecar discipline: never enters proof path; never raises;
returns dict with `tier`, `tier_rank` (1=strongest, 99=quiet),
`signals`, `candidate_names`, `note`. 20 tests cover each tier
+ noise filtering + sidecar contract + tier-ordering (strongest
wins when multiple fire).

Wiring into `arborist inspect` sidecar output + audit-line
render-tail is queued as a follow-up — sidecar itself ready.

#000025 5F Phase 1c — fixture catalog expansion
================================================

Synthetic side of all five 5F sub-batteries expanded 10 → 30:

  function       — varied claim_count, pointer_set, threshold cases
  falsification  — 13 violation tags (WARRANT_MISSING, TITLE_MISMATCH,
                   FORMAT_COLLAPSED, NO_EVIDENCE_POINTER, BARE_NAME_CLAIM,
                   LAZY_ANCHOR_DEMOTED, etc.) + 7 fail cases
  feedback-loop  — 10 chain templates × 2 cycles
  finetuning     — 20 capability transitions across all 5S/5T/5F/5R
                   sub-batteries + canonical math/logic
  formulate      — 12 lattice shapes × 2 (with deliberate fail cases)

150/150 fixtures pass through `bench-5f-*` runners.
test_session_integration.py total updated 462 → 562. Pinned
test_5f_*_runs counts updated 10 → 30 across all assertions.

Tests
=====

Full suite: 1388 passed, 36 skipped (was 1367; +21 — 20 warrant
tests + 1 ForkScore --out test).
2026-05-09 12:14:38 -04:00

31 lines
13 KiB
JSON
Raw Blame History

{"_meta": {"battery": "5f", "sub_battery": "function", "version": "v1", "task_count": 30, "notes": "Phase 1a: ten seed tasks across shape_match/pointer_set_match/threshold_on_metric evaluators. Phase 1b expands to 30+. Phase 1c (2026-05-09): expanded 10→30."}}
{"id":"5f-fn-001","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Return exactly two claim-lattice claims with evidence pointers.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"first claim","pointer_ids":["E1"]},{"claim_text":"second claim","pointer_ids":["E2"]}]},"expected":"pass"}
{"id":"5f-fn-002","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Return exactly three claims with pointers.","evaluator":"shape_match","expected_shape":{"claim_count":3,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":["E1"]},{"claim_text":"b","pointer_ids":["E2"]},{"claim_text":"c","pointer_ids":["E3"]}]},"expected":"pass"}
{"id":"5f-fn-003","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect missing pointers in claims.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":[]},{"claim_text":"b","pointer_ids":["E2"]}]},"expected":"fail"}
{"id":"5f-fn-004","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect wrong claim count.","evaluator":"shape_match","expected_shape":{"claim_count":2,"pointers_required":true},"produced_output":{"claims":[{"claim_text":"a","pointer_ids":["E1"]}]},"expected":"fail"}
{"id":"5f-fn-005","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify pointer set matches.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2","E3"]},"produced_output":{"claims":[{"claim_text":"x","pointer_ids":["E1","E2"]},{"claim_text":"y","pointer_ids":["E3"]}]},"expected":"pass"}
{"id":"5f-fn-006","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect pointer-set mismatch.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1","E2"]},"produced_output":{"claims":[{"claim_text":"x","pointer_ids":["E1","E5"]}]},"expected":"fail"}
{"id":"5f-fn-007","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify directive coverage threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"directive_coverage","threshold":0.8},"produced_output":{"directive_coverage":0.95},"expected":"pass"}
{"id":"5f-fn-008","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify strict-rate threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"strict_rate","threshold":0.5},"produced_output":{"strict_rate":0.62},"expected":"pass"}
{"id":"5f-fn-009","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Detect failed metric threshold.","evaluator":"threshold_on_metric","expected_shape":{"metric":"strict_rate","threshold":0.5},"produced_output":{"strict_rate":0.31},"expected":"fail"}
{"id":"5f-fn-010","battery":"5f","sub_battery":"function","version":"v1","carrier":"claim_lattice","domain":"qa_answer","pi_star_ref":"claim-lattice@v1","directive":"Verify pointer set with single pointer.","evaluator":"pointer_set_match","expected_shape":{"pointer_set":["E1"]},"produced_output":{"claims":[{"claim_text":"single","pointer_ids":["E1"]}]},"expected":"pass"}
{"id": "5f-fn-011", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 1 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 1, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}]}, "expected": "pass"}
{"id": "5f-fn-012", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 4 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 4, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}]}, "expected": "pass"}
{"id": "5f-fn-013", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 5 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}]}, "expected": "pass"}
{"id": "5f-fn-014", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 6 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 6, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}]}, "expected": "pass"}
{"id": "5f-fn-015", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 7 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 7, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}]}, "expected": "pass"}
{"id": "5f-fn-016", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Return exactly 8 claims with pointers.", "evaluator": "shape_match", "expected_shape": {"claim_count": 8, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "claim 1", "pointer_ids": ["E1"]}, {"claim_text": "claim 2", "pointer_ids": ["E2"]}, {"claim_text": "claim 3", "pointer_ids": ["E3"]}, {"claim_text": "claim 4", "pointer_ids": ["E4"]}, {"claim_text": "claim 5", "pointer_ids": ["E5"]}, {"claim_text": "claim 6", "pointer_ids": ["E6"]}, {"claim_text": "claim 7", "pointer_ids": ["E7"]}, {"claim_text": "claim 8", "pointer_ids": ["E8"]}]}, "expected": "pass"}
{"id": "5f-fn-017", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (1 vs expected 2).", "evaluator": "shape_match", "expected_shape": {"claim_count": 2, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}]}, "expected": "fail"}
{"id": "5f-fn-018", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (2 vs expected 3).", "evaluator": "shape_match", "expected_shape": {"claim_count": 3, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}, {"claim_text": "x", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-019", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Detect wrong claim count (3 vs expected 5).", "evaluator": "shape_match", "expected_shape": {"claim_count": 5, "pointers_required": true}, "produced_output": {"claims": [{"claim_text": "x", "pointer_ids": ["E1"]}, {"claim_text": "x", "pointer_ids": ["E2"]}, {"claim_text": "x", "pointer_ids": ["E3"]}]}, "expected": "fail"}
{"id": "5f-fn-020", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3", "E4"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1", "E2"]}, {"claim_text": "c2", "pointer_ids": ["E3", "E4"]}]}, "expected": "pass"}
{"id": "5f-fn-021", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1", "E2"]}]}, "expected": "pass"}
{"id": "5f-fn-022", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-023", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Pointer-set comparison.", "evaluator": "pointer_set_match", "expected_shape": {"pointer_set": ["E1", "E2", "E3"]}, "produced_output": {"claims": [{"claim_text": "c1", "pointer_ids": ["E1"]}, {"claim_text": "c2", "pointer_ids": ["E2"]}]}, "expected": "fail"}
{"id": "5f-fn-024", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.5}, "produced_output": {"strict_rate": 0.7}, "expected": "pass"}
{"id": "5f-fn-025", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.5}, "produced_output": {"strict_rate": 0.45}, "expected": "fail"}
{"id": "5f-fn-026", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on directive_coverage.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "directive_coverage", "threshold": 0.9}, "produced_output": {"directive_coverage": 0.99}, "expected": "pass"}
{"id": "5f-fn-027", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on directive_coverage.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "directive_coverage", "threshold": 0.95}, "produced_output": {"directive_coverage": 0.92}, "expected": "fail"}
{"id": "5f-fn-028", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on hybrid_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "hybrid_rate", "threshold": 0.3}, "produced_output": {"hybrid_rate": 0.32}, "expected": "pass"}
{"id": "5f-fn-029", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate (sub-floor).", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.55}, "produced_output": {"strict_rate": 0.4}, "expected": "fail"}
{"id": "5f-fn-030", "battery": "5f", "sub_battery": "function", "version": "v1", "carrier": "claim_lattice", "domain": "qa_answer", "pi_star_ref": "claim-lattice@v1", "directive": "Threshold on strict_rate.", "evaluator": "threshold_on_metric", "expected_shape": {"metric": "strict_rate", "threshold": 0.6}, "produced_output": {"strict_rate": 0.6}, "expected": "pass"}
Reference in a new issue View git blame Copy permalink