engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000048 closed; #000049 opened — the attribution-aware verifier is its own ticket

Browse source 
fox 2026-05-12: the attribution-aware path (#000048's deferred §2.3 —
closing the 2 recombination over-grounds in falsification-hard) is its
own ticket, not a #000048 phase. So:

#000048 → closed (at 2.1 + 2.4). Steps 2.1 + 2.4 landed 2026-05-11
(12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate
mis-segments → formulate-hard 12/12, falsification-hard 10/12; each
bench-gated, no STRICT-rate regression). Step 2.2 (single-clause-
containment paraphrase check) attempted + reverted — recombination-
vs-summary isn't lexical (§5 "What we learned"). The 2 residual
falsification-hard fixtures (hard-003 Mercury, hard-005 Einstein)
stand as a documented marker of where the lexical verifier stops.
Header + §5 Closure + §2.3 updated; cross-refs in #000046 / #000012
§8 / TICKETS.md repointed from "#000048 §2.3" to "#000049".

#000049 opened (doc-only, awaiting go/no-go) — "Attribution-aware
grounding check (the recombination boundary)". The recombination
class needs an attribution / dependency-parse or mini-NLI check
(distinguishing "Mercury is the largest" against "Jupiter is the
largest; Mercury is the smallest" from a legit cross-sentence
summary). Options: 2.1 hand-rolled dependency-attribution heuristic
(no model, brittle — same threshold-can't-separate problem one rung
up); 2.2 small purpose-built NLI model ([nli] extra, policy-gated,
off-by-default, bench-gated — the right capability, but forces the
"is a fixed NLI model an LLM-judge?" discipline call + a model
dependency + a non-determinism surface to pin); 2.3 do nothing (the
2 fixtures are a boundary marker, no observed real-traffic harm).
Recommends 2.3 until real-traffic recombination over-grounds show up,
then 2.2 *if* fox rules a fixed NLI model is acceptable in the
proof-adjacent path; the first decision the ticket needs is that
discipline question. Next ID 000049 → 000050. #000048 follow-up.
Doc-only — no code change.
This commit is contained in:
russell@unturf.com 2026-05-12 08:31:57 -04:00
parent 2ad2dceada
commit 20f6061f83
No known key found for this signature in database
5 changed files with 253 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

7
docs/TICKETS.md
View file

@ -92,9 +92,10 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **steps 2.1 + 2.4 landed 2026-05-11; step 2.2 attempted + reverted.** 2.1: `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`) → `falsification-hard` 6/12 → 10/12. 2.4: `parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` split kept only when every segment is well-pointered; leading colon-header dropped; wrapped-bullet join) → `formulate-hard` 4/12 → 12/12 (at ceiling). Each bench-gated (`make bench-qa` n=3×75×3 — 0 / 7-lattice-cells-touched-both-correct; all deltas within the 5-pp floor). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. 2.2 (single-clause-containment paraphrase check): catches the 2 recombination fixtures but also rejects legit cross-sentence summaries (the Batman case) with no threshold separating the two — recombination-vs-summary isn't lexical; reverted. **Recommendation: close at 2.1+2.4** (12 of 16 residual items closed; the 2 `falsification-hard` recombination fixtures a documented marker; an attribution-aware verifier = §2.3, its own ticket if ever). #000046 follow-up; awaiting fox's close-or-keep-open call | 2026-05-11 | — |
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · awaiting go/no-go · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). First decision = the discipline question: may a small fixed purpose-built NLI/entailment *model* influence `audit_mode`? (vs the "no LLM-as-judge" rule). Recommends Option 2.3 (do nothing — the 2 fixtures are a boundary marker) until real-traffic recombination-over-grounds show up, then Option 2.2 (`[nli]` extra, policy-gated, off-by-default, bench-gated) *if* fox rules a fixed NLI model is acceptable. #000048 follow-up | 2026-05-12 | — |
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — |
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, #000048 §2.3, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, now tracked as **#000049**, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 closed | 2026-05-11 | — |
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
@ -143,4 +144,4 @@ Newest first. Update on every open/close.
## Next ID
`000049`
`000050`