engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix burn-kindergarten 0-second flake; expose --repair on query (off by default)

Browse source 
burn-kindergarten flake fix:
test_burn_kindergarten_zero_seconds_burns_everything failed under
full-suite ordering when the wall-clock second rolled over between
seeding the row & running burn-kindergarten. Old impl computed
`cutoff = now - 0 = now` and required `created_at >= now`, so a row
seeded at second T became invisible to a burn check that ran at T+1.

The verb's docstring is "0-second window = burn every live row".
Fix the impl to match: when kindergarten_seconds <= 0, drop the time
gate entirely (burn all live records). Non-zero windows keep the
existing cutoff-based filter unchanged.

Repair CLI:
`aborist query --repair` enables the mechanical repair pass after
first verify. `--repair-reprompts N` adds the optional re-prompt
tier (default 0 = off; mechanical-only when --repair is set without
--repair-reprompts). Both flags are off by default so `make query`
stays single-shot unless the operator opts in.

Makefile: `make query REPAIR=1` flips on mechanical repair;
`make query REPAIR=1 REPROMPTS=1` adds one re-prompt iteration.
Both knobs off by default to preserve existing behavior.

485 tests pass under full-suite ordering (was 484 + 1 flake).
This commit is contained in:
russell@unturf.com 2026-04-29 19:57:24 -04:00
parent 9930fc7f1d
commit 304dadfb71
No known key found for this signature in database
2 changed files with 56 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Makefile
View file

@ -155,11 +155,11 @@ burn-kindergarten: bootstrap ## bust providence rows < SECONDS old [SECONDS=3600
# Multi-source RAG query against the shard cluster.
# Usage: make query Q="What is anarcho-capitalism?"
QUERY_TOP_K ?= 8
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1]; sources picked across shards
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N]; sources picked across shards
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1]"; exit 2; \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N]"; exit 2; \
fi
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) "$(Q)"
$(ABORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) "$(Q)"
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1]
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \

63
aborist/cli.py
View file

@ -343,12 +343,20 @@ def _cmd_query(args: argparse.Namespace) -> int:
)
single_db = None if shards_dir else args.db
# Apply per-call policy overrides (question_dedup) on top of the
# default. fidelity is a function-level kwarg, not in the policy.
# Apply per-call policy overrides (question_dedup, repair) on top of
# the default. fidelity is a function-level kwarg, not in the policy.
from aborist.qa.query import DEFAULT_QUERY_POLICY
call_policy = dict(DEFAULT_QUERY_POLICY)
if getattr(args, "question_dedup", None):
call_policy["question_dedup"] = args.question_dedup
if getattr(args, "repair", False):
# Mechanical-only repair when --repair is set; --repair-reprompts
# adds the optional re-prompt tier on top. Both default off so
# `make query` stays single-shot unless a knob is flipped.
call_policy["repair_enabled"] = True
call_policy["repair_max_reprompts"] = max(
0, int(getattr(args, "repair_reprompts", 0))
)
result = query(
question=args.question,
@ -1057,7 +1065,15 @@ def _cmd_burn_kindergarten(args: argparse.Namespace) -> int:
from aborist.store import discover_shards
now = int(_time.time())
cutoff = now - max(0, args.kindergarten_seconds)
# `kindergarten_seconds <= 0` means "no time gate — burn every live
# row" per the verb's docstring. The earlier `cutoff = now - 0`
# treated 0 as a same-second-only window, which made
# test_burn_kindergarten_zero_seconds_burns_everything timing-flaky:
# if the wall-clock second rolled over between seed and burn,
# cutoff > seed.created_at and nothing matched.
kindergarten_seconds = max(0, args.kindergarten_seconds)
no_time_gate = kindergarten_seconds == 0
cutoff = 0 if no_time_gate else now - kindergarten_seconds
shards_dir = Path(args.global_shards_dir) if args.global_shards_dir else None
single_db = Path(args.db) if args.db else None
paths: list[Path] = (
@ -1074,13 +1090,21 @@ def _cmd_burn_kindergarten(args: argparse.Namespace) -> int:
for sp in paths:
c = connect(sp)
try:
rows = c.execute(
"SELECT cache_key, created_at, audit_mode "
"FROM providence_cache "
"WHERE created_at >= ? AND falsification_state = 'live' "
"ORDER BY created_at DESC",
(cutoff,),
).fetchall()
if no_time_gate:
rows = c.execute(
"SELECT cache_key, created_at, audit_mode "
"FROM providence_cache "
"WHERE falsification_state = 'live' "
"ORDER BY created_at DESC"
).fetchall()
else:
rows = c.execute(
"SELECT cache_key, created_at, audit_mode "
"FROM providence_cache "
"WHERE created_at >= ? AND falsification_state = 'live' "
"ORDER BY created_at DESC",
(cutoff,),
).fetchall()
finally:
c.close()
for r in rows:
@ -2856,6 +2880,25 @@ def build_parser() -> argparse.ArgumentParser:
"audit event."
),
)
query_cmd.add_argument(
"--repair", action="store_true",
help=(
"enable mechanical repair after first verify (off by "
"default). When the verdict is HYBRID/UNGROUNDED, applies "
"synthetic_elision split, trailing_artifact trim, and "
"no_overlap drop deterministically; persists the repaired "
"answer with a providence_repair audit event."
),
)
query_cmd.add_argument(
"--repair-reprompts", dest="repair_reprompts", type=int, default=0,
help=(
"max LLM re-prompt iterations after mechanical repair "
"(default 0 = no re-prompt). Each iteration sends a feedback "
"turn naming failed quotes; the model is asked to rewrite "
"using only verbatim citations. Requires --repair."
),
)
query_cmd.set_defaults(func=_cmd_query)
inspect_cmd = sub.add_parser(