qa/corpus: shared title-boost (extras penalty) lifted into run_query
The Hank-Scorpio-vs-Mr.-Burns problem: body BM25 alone outranks the
canonical primary-source article ("Homer Simpson") below sibling
articles that incidentally mention the same tokens. The sidecar
pipeline already had title-boost-with-extras-penalty; lift it into
a shared utility that the orchestrator applies to ANY adapter's
fts_body output.
arborist/qa/corpus.py:apply_title_boost(hits, query, *,
higher_is_better)
Stems both sides (possessive + plural collapse), numeral-expands
(7↔VII), accent-folds (é→e). Effective bonus per hit:
max(0, overlap - extras/2) * boost
where extras = title tokens NOT in query. Score direction honors
each adapter's convention (BM25 negative → subtract; sidecar BM25
positive → add).
arborist/qa/corpus_query.py:run_query
Now oversamples fts_body by 4× and reranks via apply_title_boost
before slicing to top_k. Without oversampling the right primary
can sit at rank 7-15 in the body BM25 output and get cut before
the rerank sees it.
bench/three_way_bench.py: NEW — drives same fixture through
- `arborist query` (legacy 2000-line pipeline)
- `arborist corpus-query` (new, local shards via Corpus + run_query)
- `arborist cloud query` (new, sidecar via Corpus + run_query)
prints 3-column table + flags primary-source disagreements.
Smoke fixture (5 questions, all paths):
4/5 all-paths agree on primary source (was 2/5 pre-fix)
1/5 disagrees: dinosaur-extinction
legacy: UNGROUNDED · Edwina (children's book) ← wrong
corpus: STRICT · Dinosaur (main article) ← right
cloud: HYBRID · Edwina (children's book) ← wrong
The new title-boost lifted corpus-query above the legacy here;
cloud still picks Edwina because SidecarReader.search applies
title-boost INTERNALLY (it predates the shared util), so the
extras-penalty stacks weirdly when run_query applies it again.
Next fix: disable internal boost in SidecarReader once shared
util is the single source of truth.
26 corpus/wallet/sidecar tests still green.
This commit is contained in:
parent
510e05b979
commit
2b8777c947
6 changed files with 319 additions and 4 deletions
|
|
@ -41,6 +41,97 @@ class NotSupportedError(NotImplementedError):
|
|||
backend. The orchestrator should catch and skip — never propagate."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared retrieval-quality post-processor: title-boost with extras penalty.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def apply_title_boost(
|
||||
hits: list["Hit"],
|
||||
query: str,
|
||||
*,
|
||||
higher_is_better: bool,
|
||||
boost: float = 8.0,
|
||||
) -> list["Hit"]:
|
||||
"""Re-rank hits by query-vs-title token overlap, penalized by extras.
|
||||
|
||||
Without this, body BM25 alone consistently outranks the canonical
|
||||
primary-source article ("Homer Simpson") below sibling articles
|
||||
("You Only Move Twice", "Fat Tony") that incidentally mention the
|
||||
same tokens. The sidecar pipeline already does this on construction;
|
||||
here we lift it into a shared utility so SqliteShardCorpus +
|
||||
SidecarBucketCorpus + future adapters all benefit through the
|
||||
same orchestrator step.
|
||||
|
||||
Score direction:
|
||||
- higher_is_better=True (sidecar BM25): boost ADDED to score
|
||||
- higher_is_better=False (FTS5 BM25): boost SUBTRACTED from score
|
||||
|
||||
Effective boost magnitude:
|
||||
max(0, overlap - extras/2) * boost
|
||||
|
||||
where overlap = |query_stems ∩ title_stems|
|
||||
and extras = |title_stems - query_stems|
|
||||
Stems strip possessive apostrophes + trailing-s plurals; both
|
||||
sides are numeral-expanded (7↔VII) and accent-folded (é→e).
|
||||
"""
|
||||
if not hits or not query.strip() or boost <= 0:
|
||||
return hits
|
||||
# Lazy-import: sidecar carries the tokenizer + numeral_expand;
|
||||
# importing at module top would loop on the from-sidecar imports.
|
||||
from arborist.wallet.sidecar import (
|
||||
_WORD_RE, STOPWORDS,
|
||||
fold_accents, numeral_expand, tokenize_text,
|
||||
)
|
||||
|
||||
def _stem(t: str) -> str:
|
||||
t = t.replace("'", "").replace("’", "")
|
||||
if len(t) > 4 and t.endswith("s") and not t.endswith("ss"):
|
||||
return t[:-1]
|
||||
return t
|
||||
|
||||
query_stems = numeral_expand({_stem(t) for t in tokenize_text(query)})
|
||||
if not query_stems:
|
||||
return hits
|
||||
|
||||
rescored: list[Hit] = []
|
||||
for h in hits:
|
||||
title = fold_accents((h.title or "").lower())
|
||||
raw_title_tokens: set[str] = set()
|
||||
for tok in _WORD_RE.findall(title):
|
||||
if tok in STOPWORDS:
|
||||
continue
|
||||
if len(tok) <= 1 and not tok.isdigit():
|
||||
continue
|
||||
raw_title_tokens.add(_stem(tok))
|
||||
title_tokens = numeral_expand(raw_title_tokens)
|
||||
overlap = len(query_stems & title_tokens)
|
||||
if not overlap:
|
||||
rescored.append(h)
|
||||
continue
|
||||
extras = len(title_tokens - query_stems)
|
||||
effective = max(0.0, overlap - extras / 2.0)
|
||||
if effective <= 0:
|
||||
rescored.append(h)
|
||||
continue
|
||||
delta = effective * boost
|
||||
new_score = h.score + delta if higher_is_better else h.score - delta
|
||||
rescored.append(Hit(
|
||||
document_root=h.document_root,
|
||||
document_uri=h.document_uri,
|
||||
title=h.title,
|
||||
score=new_score,
|
||||
shard_id=h.shard_id,
|
||||
extras={**h.extras, "title_boost_delta": round(delta, 3)},
|
||||
))
|
||||
# Re-sort by score in the adapter's convention.
|
||||
rescored.sort(
|
||||
key=lambda h: h.score,
|
||||
reverse=higher_is_better,
|
||||
)
|
||||
return rescored
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Hit:
|
||||
"""One retrieval result, backend-agnostic.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ from typing import Optional
|
|||
|
||||
from arborist.compress import unpack_chunk
|
||||
from arborist.qa.client import ChatClient
|
||||
from arborist.qa.corpus import Corpus, NotSupportedError
|
||||
from arborist.qa.corpus import Corpus, NotSupportedError, apply_title_boost
|
||||
from arborist.qa.evidence import (
|
||||
build_evidence_map,
|
||||
evidence_map_by_pointer_id,
|
||||
|
|
@ -78,13 +78,23 @@ def run_query(
|
|||
t_start = _time.time()
|
||||
|
||||
# 1. Retrieve — fts_body is the only route currently shared across
|
||||
# adapters. Future: add fts_title / fts_phrase / core_keyword
|
||||
# with NotSupported skip-and-merge logic.
|
||||
# adapters. Oversample 4× so the post-retrieval title-boost has
|
||||
# more candidates to rerank (without oversampling, the right
|
||||
# primary-source article can be at rank 7-15 in body BM25 output
|
||||
# and get cut before reranking sees it). Future: add fts_title /
|
||||
# fts_phrase / core_keyword with NotSupported skip-and-merge.
|
||||
ts = _time.time()
|
||||
try:
|
||||
hits = corpus.fts_body(question, limit=top_k)
|
||||
hits = corpus.fts_body(question, limit=top_k * 4)
|
||||
except NotSupportedError:
|
||||
hits = []
|
||||
# Shared title-boost rerank — lifts "Homer Simpson" main article
|
||||
# above "You Only Move Twice" sibling on the homer query, etc.
|
||||
# Same logic for SqliteShardCorpus + SidecarBucketCorpus.
|
||||
hits = apply_title_boost(
|
||||
hits, question,
|
||||
higher_is_better=getattr(corpus, "higher_is_better", False),
|
||||
)[:top_k]
|
||||
timings["search"] = _time.time() - ts
|
||||
|
||||
if not hits:
|
||||
|
|
|
|||
5
bench/cloud_vs_local_results/2026-05-31T12-25-59Z.jsonl
Normal file
5
bench/cloud_vs_local_results/2026-05-31T12-25-59Z.jsonl
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
{"question": "when did the soviet union dissolve?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 6.38, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 47.45, "error": null}, "regression": false, "reason": "ok"}
|
||||
{"question": "where is mount kilimanjaro located?", "local": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 5.08, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 54.33, "error": null}, "regression": false, "reason": "ok"}
|
||||
{"question": "who painted the mona lisa?", "local": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa", "used": true}, "elapsed_s": 5.89, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa", "used": true}, "elapsed_s": 47.75, "error": null}, "regression": false, "reason": "ok"}
|
||||
{"question": "who were the original seven mercury astronauts?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven", "used": true}, "elapsed_s": 7.0, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_quotes": 8, "n_verified": 8, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven", "used": true}, "elapsed_s": 53.95, "error": null}, "regression": true, "reason": "audit_mode dropped STRICT → HYBRID"}
|
||||
{"question": "why did the dinosaurs go extinct?", "local": {"audit_mode": "UNGROUNDED", "n_quotes": 1, "n_verified": 0, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct", "used": false}, "elapsed_s": 5.81, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_quotes": 4, "n_verified": 1, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct", "used": true}, "elapsed_s": 58.5, "error": null}, "regression": false, "reason": "ok"}
|
||||
2
bench/cloud_vs_local_results/2026-05-31T12-30-06Z.jsonl
Normal file
2
bench/cloud_vs_local_results/2026-05-31T12-30-06Z.jsonl
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
{"question": "when did the soviet union dissolve?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 4.64, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union", "used": true}, "elapsed_s": 56.16, "error": null}, "regression": false, "reason": "ok"}
|
||||
{"question": "where is mount kilimanjaro located?", "local": {"audit_mode": "STRICT", "n_quotes": 1, "n_verified": 1, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 3.71, "error": null}, "cloud": {"audit_mode": "STRICT", "n_quotes": 2, "n_verified": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro", "used": true}, "elapsed_s": 56.46, "error": null}, "regression": false, "reason": "ok"}
|
||||
202
bench/three_way_bench.py
Normal file
202
bench/three_way_bench.py
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
"""3-way bench: legacy local vs corpus-query vs cloud-query.
|
||||
|
||||
Goal: every fixture question produces the SAME primary-source citation
|
||||
across all three paths. Diverging primary-source picks are real
|
||||
regressions even when audit_mode happens to agree.
|
||||
|
||||
legacy arborist --shards-dir … query --json --burn --answer-mode claim_lattice ...
|
||||
corpus-query arborist --shards-dir … corpus-query --json ... (Corpus → run_query)
|
||||
cloud-query arborist cloud query --bucket-url <manifest> --json ... (Sidecar → run_query)
|
||||
|
||||
JSONL output: one row per question with all three path results +
|
||||
agreement flags so iterations can compare across bench runs.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as _dt
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
ARBORIST = REPO / ".venv" / "bin" / "arborist"
|
||||
DEFAULT_SHARDS = Path.home() / ".arborist" / "shards"
|
||||
DEFAULT_BUCKET = (
|
||||
"https://nyc3.digitaloceanspaces.com/arborist/clones/manifest-sidecar.json"
|
||||
)
|
||||
DEFAULT_QWEN_ENDPOINT = "https://qwen.ai.unturf.com/v1"
|
||||
DEFAULT_QWEN_MODEL = "Qwen3.6-27B-UD-Q4_K_XL.gguf"
|
||||
|
||||
|
||||
def _load_fixture(path: Path) -> list[str]:
|
||||
out = []
|
||||
for line in path.read_text().splitlines():
|
||||
s = line.strip()
|
||||
if not s or s.startswith("#"):
|
||||
continue
|
||||
out.append(s)
|
||||
return out
|
||||
|
||||
|
||||
def _run(cmd: list[str], *, timeout_s: int) -> dict:
|
||||
t0 = time.time()
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=timeout_s,
|
||||
env={**os.environ, "ARBORIST_PROGRESS": "0"},
|
||||
)
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"_error": f"timeout {timeout_s}s", "_elapsed_s": timeout_s}
|
||||
dt = time.time() - t0
|
||||
if proc.returncode != 0:
|
||||
return {"_error": f"exit {proc.returncode}",
|
||||
"_stderr": proc.stderr[:300], "_elapsed_s": round(dt, 2)}
|
||||
try:
|
||||
d = json.loads(proc.stdout)
|
||||
except json.JSONDecodeError as e:
|
||||
return {"_error": f"parse: {e}", "_stdout_head": proc.stdout[:200],
|
||||
"_elapsed_s": round(dt, 2)}
|
||||
d["_elapsed_s"] = round(dt, 2)
|
||||
return d
|
||||
|
||||
|
||||
def _legacy(q, *, shards_dir):
|
||||
return _run([
|
||||
str(ARBORIST), "--shards-dir", str(shards_dir),
|
||||
"query", "--top-k", "8", "--json", "--burn",
|
||||
"--answer-mode", "claim_lattice", q,
|
||||
], timeout_s=120)
|
||||
|
||||
|
||||
def _corpus(q, *, shards_dir, endpoint, model):
|
||||
return _run([
|
||||
str(ARBORIST), "--shards-dir", str(shards_dir),
|
||||
"corpus-query", q,
|
||||
"--top-k", "4", "--json",
|
||||
"--endpoint", endpoint, "--model", model,
|
||||
], timeout_s=120)
|
||||
|
||||
|
||||
def _cloud(q, *, bucket_url, endpoint, model):
|
||||
return _run([
|
||||
str(ARBORIST), "cloud", "query", q,
|
||||
"--bucket-url", bucket_url,
|
||||
"--endpoint", endpoint, "--model", model,
|
||||
"--top-k", "4", "--json",
|
||||
], timeout_s=240)
|
||||
|
||||
|
||||
def _primary(r: dict) -> dict:
|
||||
srcs = r.get("sources") or []
|
||||
if not srcs:
|
||||
return {"title": "", "uri": ""}
|
||||
pri = next(
|
||||
(s for s in srcs if s.get("source_role") == "primary_answer_source"),
|
||||
srcs[0],
|
||||
)
|
||||
return {"title": (pri.get("title") or "")[:55],
|
||||
"uri": pri.get("document_uri") or ""}
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument(
|
||||
"--fixture", type=Path,
|
||||
default=REPO / "bench" / "qa_questions_smoke.txt",
|
||||
)
|
||||
p.add_argument("--shards-dir", type=Path, default=DEFAULT_SHARDS)
|
||||
p.add_argument("--bucket-url", default=DEFAULT_BUCKET)
|
||||
p.add_argument("--endpoint", default=DEFAULT_QWEN_ENDPOINT)
|
||||
p.add_argument("--model", default=DEFAULT_QWEN_MODEL)
|
||||
p.add_argument(
|
||||
"--out-dir", type=Path,
|
||||
default=REPO / "bench" / "three_way_results",
|
||||
)
|
||||
p.add_argument("--skip-legacy", action="store_true")
|
||||
p.add_argument("--skip-corpus", action="store_true")
|
||||
p.add_argument("--skip-cloud", action="store_true")
|
||||
args = p.parse_args()
|
||||
|
||||
questions = _load_fixture(args.fixture)
|
||||
args.out_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = _dt.datetime.now(_dt.timezone.utc).strftime("%Y-%m-%dT%H-%M-%SZ")
|
||||
out_path = args.out_dir / f"{stamp}.jsonl"
|
||||
|
||||
print(f"# {len(questions)} questions from {args.fixture}", file=sys.stderr)
|
||||
print(f"# shards: {args.shards_dir}", file=sys.stderr)
|
||||
print(f"# bucket: {args.bucket_url}", file=sys.stderr)
|
||||
print(f"# llm: {args.endpoint} / {args.model}", file=sys.stderr)
|
||||
print(file=sys.stderr)
|
||||
|
||||
n_full_agreement = 0
|
||||
n_source_disagree = 0
|
||||
|
||||
with open(out_path, "w") as f:
|
||||
for i, q in enumerate(questions, 1):
|
||||
print(f"[{i}/{len(questions)}] {q}", file=sys.stderr)
|
||||
legacy = {} if args.skip_legacy else _legacy(q, shards_dir=args.shards_dir)
|
||||
corpus = {} if args.skip_corpus else _corpus(
|
||||
q, shards_dir=args.shards_dir,
|
||||
endpoint=args.endpoint, model=args.model,
|
||||
)
|
||||
cloud = {} if args.skip_cloud else _cloud(
|
||||
q, bucket_url=args.bucket_url,
|
||||
endpoint=args.endpoint, model=args.model,
|
||||
)
|
||||
|
||||
row = {"question": q}
|
||||
for label, r in (("legacy", legacy), ("corpus", corpus), ("cloud", cloud)):
|
||||
row[label] = {
|
||||
"audit_mode": r.get("audit_mode"),
|
||||
"n_verified": r.get("n_verified"),
|
||||
"n_quotes": r.get("n_quotes"),
|
||||
"primary": _primary(r),
|
||||
"elapsed_s": r.get("_elapsed_s"),
|
||||
"error": r.get("_error"),
|
||||
}
|
||||
# Agreement: same primary URI across all PRESENT paths
|
||||
uris = {
|
||||
lbl: row[lbl]["primary"]["uri"]
|
||||
for lbl in ("legacy", "corpus", "cloud")
|
||||
if row[lbl]["primary"]["uri"]
|
||||
}
|
||||
distinct = set(uris.values())
|
||||
row["agreement"] = {
|
||||
"n_paths_with_source": len(uris),
|
||||
"distinct_sources": len(distinct),
|
||||
"all_agree": len(distinct) <= 1,
|
||||
}
|
||||
if row["agreement"]["all_agree"]:
|
||||
n_full_agreement += 1
|
||||
else:
|
||||
n_source_disagree += 1
|
||||
|
||||
f.write(json.dumps(row, ensure_ascii=False) + "\n")
|
||||
f.flush()
|
||||
|
||||
for label in ("legacy", "corpus", "cloud"):
|
||||
r = row[label]
|
||||
badge = " ←" if r["primary"]["uri"] and len(distinct) > 1 and r["primary"]["uri"] != min(uris.values()) else ""
|
||||
err = f" ERR={r['error']}" if r["error"] else ""
|
||||
print(f" {label:7s} {(r['audit_mode'] or '?'):<25} "
|
||||
f"{r['primary']['title']}{badge}{err}",
|
||||
file=sys.stderr)
|
||||
if not row["agreement"]["all_agree"]:
|
||||
print(f" ⚠ SOURCE DISAGREEMENT: {len(distinct)} distinct primaries",
|
||||
file=sys.stderr)
|
||||
print(file=sys.stderr)
|
||||
|
||||
print(f"=== SUMMARY ===", file=sys.stderr)
|
||||
print(f" {len(questions)} questions", file=sys.stderr)
|
||||
print(f" {n_full_agreement} all-paths agree on primary source", file=sys.stderr)
|
||||
print(f" {n_source_disagree} paths-disagree", file=sys.stderr)
|
||||
print(f" results: {out_path}", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
5
bench/three_way_results/2026-05-31T12-33-40Z.jsonl
Normal file
5
bench/three_way_results/2026-05-31T12-33-40Z.jsonl
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
{"question": "when did the soviet union dissolve?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 4.51, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 40.06, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Soviet Union", "uri": "https://en.wikipedia.org/wiki/Soviet_Union"}, "elapsed_s": 49.53, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
|
||||
{"question": "where is mount kilimanjaro located?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 3.67, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 46.7, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mount Kilimanjaro", "uri": "https://en.wikipedia.org/wiki/Mount_Kilimanjaro"}, "elapsed_s": 52.95, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
|
||||
{"question": "who painted the mona lisa?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 4.06, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 11.97, "error": null}, "cloud": {"audit_mode": "STRICT", "n_verified": 2, "n_quotes": 2, "primary": {"title": "Mona Lisa", "uri": "https://en.wikipedia.org/wiki/Mona_Lisa"}, "elapsed_s": 46.6, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
|
||||
{"question": "who were the original seven mercury astronauts?", "legacy": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 7.58, "error": null}, "corpus": {"audit_mode": "HYBRID", "n_verified": 4, "n_quotes": 4, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 33.99, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_verified": 8, "n_quotes": 8, "primary": {"title": "Mercury Seven", "uri": "https://en.wikipedia.org/wiki/Mercury_Seven"}, "elapsed_s": 55.25, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 1, "all_agree": true}}
|
||||
{"question": "why did the dinosaurs go extinct?", "legacy": {"audit_mode": "UNGROUNDED", "n_verified": 0, "n_quotes": 1, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct"}, "elapsed_s": 7.05, "error": null}, "corpus": {"audit_mode": "STRICT", "n_verified": 1, "n_quotes": 1, "primary": {"title": "Dinosaur", "uri": "https://en.wikipedia.org/wiki/Dinosaur"}, "elapsed_s": 42.81, "error": null}, "cloud": {"audit_mode": "HYBRID", "n_verified": 1, "n_quotes": 4, "primary": {"title": "Edwina, the Dinosaur Who Didn't Know She Was Extinct", "uri": "https://en.wikipedia.org/wiki/Edwina,_the_Dinosaur_Who_Didn't_Know_She_Was_Extinct"}, "elapsed_s": 115.13, "error": null}, "agreement": {"n_paths_with_source": 3, "distinct_sources": 2, "all_agree": false}}
|
||||
Loading…
Add table
Add a link
Reference in a new issue