russell@unturf.com
8d6961fcc1
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py
154 lines
6.4 KiB
Text
154 lines
6.4 KiB
Text
# arborist QA-quality benchmark question set.
|
|
#
|
|
# One question per line. `#`-prefixed lines and blank lines ignored.
|
|
# Designed for the 2003-05-16 Wikipedia cur snapshot — questions must
|
|
# resolve against ~2003 article content. Stress-tests across the
|
|
# failure-mode shapes the verifier needs to handle.
|
|
|
|
# narrow factoid — well-anchored single-fact questions
|
|
what is the capital of france?
|
|
who wrote GNU linux?
|
|
when was a programming language named Python created?
|
|
who painted the mona lisa?
|
|
a bridge between new london & groton?
|
|
who wrote the play hamlet?
|
|
what is the chemical symbol for gold?
|
|
|
|
# broad descriptive — encyclopedic shape, prone to mode-collapse
|
|
tell me about connecticut
|
|
tell me about the C programming language
|
|
tell me about method man?
|
|
tell me all there is to know about york england?
|
|
tell me about the roman empire
|
|
describe the structure of DNA
|
|
|
|
# under-specified "all" — the word "all" reads to Hermes-3-8B as
|
|
# license to enumerate every adjacent fact in training prior, which
|
|
# in claim_lattice_pointer mode degrades to free-form prose with zero
|
|
# `[E\d+]` tags (FORMAT_COLLAPSED soft-demote, 2026-05-02). Stronger
|
|
# models (Qwen / GPT-4 family) plausibly recover format discipline
|
|
# under the same prompt — bench coverage of this shape lets us
|
|
# measure cross-model resilience. Ticket #000008 (broad-quantifier
|
|
# preflight guard) proposes upstream classification + per-model
|
|
# claim ceiling.
|
|
winners of all major sports?
|
|
|
|
# bounded universals — finite, corpus-known answer sets. Ticket
|
|
# #000008 §10.1 splits broad universals into bounded vs unbounded.
|
|
# These should classify ALL but with `scope_bound_hint: "bounded"`,
|
|
# meaning --reject-broad does NOT reject and the cap is the natural
|
|
# bound. Without these fixtures, the bounded-vs-unbounded distinction
|
|
# has no live bench coverage.
|
|
name all members of the beatles
|
|
list all planets in the solar system
|
|
|
|
# entity list — invites lazy-anchor on a magnet chunk
|
|
what dinosaurs were in the first jurassic park film?
|
|
who are the members of the beatles?
|
|
name simpsons family members including pets?
|
|
list of obelisk in connecticut
|
|
what are the planets of our solar system?
|
|
who were the original seven mercury astronauts?
|
|
|
|
# relationship / multi-fact
|
|
who is supermans girlfriend?
|
|
who is bilbo baggins's nephew?
|
|
what is the relationship between linux & unix?
|
|
who is veronica ballestrini & what month was she born?
|
|
who is luke skywalker's father and sister?
|
|
|
|
# comparison — multi-entity, prone to attribution drift
|
|
what's the difference between linux and bsd?
|
|
how does intel compare to amd?
|
|
what is the difference between http and ftp?
|
|
mac vs windows for software development
|
|
|
|
# niche / partial — corpus may be thin
|
|
what is the boltzmann constant?
|
|
who invented the doppler effect?
|
|
|
|
# date / time — when-questions stress year/date grounding
|
|
when did world war 2 end?
|
|
what year did the berlin wall fall?
|
|
when was the first moon landing?
|
|
when did the soviet union dissolve?
|
|
in what year was the magna carta signed?
|
|
|
|
# quantity — numeric grounding, prone to confabulated digits
|
|
how many states are in the united states?
|
|
how many bones are in the adult human body?
|
|
what is the population of japan?
|
|
how many wives did henry the eighth have?
|
|
how many moons does jupiter have?
|
|
|
|
# geographic — where / what country / location grounding
|
|
where is mount kilimanjaro located?
|
|
what country is the city of prague in?
|
|
in which ocean is madagascar?
|
|
where does the nile river begin?
|
|
what continent is egypt on?
|
|
|
|
# cause / effect — why-questions, narrative grounding
|
|
why did the titanic sink?
|
|
what caused the chernobyl disaster?
|
|
why did the dinosaurs go extinct?
|
|
what triggered world war 1?
|
|
|
|
# entity disambiguation — common names that collide with many entities
|
|
who is michael jordan?
|
|
who is george bush?
|
|
who is john smith of jamestown?
|
|
what is the matrix?
|
|
who is paul of tarsus?
|
|
|
|
# synonym / paraphrase robustness — same fact, different phrasing.
|
|
# Pairs probe whether equivalence_class question dedup collapses these
|
|
# at write & whether retrieval grounds them identically.
|
|
who founded microsoft?
|
|
who is the founder of microsoft?
|
|
when was the eiffel tower built?
|
|
what year was the eiffel tower constructed?
|
|
|
|
# leading / forensic — probe whether retrieval surfaces the right
|
|
# fact when the question's premise contradicts the popular narrative.
|
|
# Elevation question expected answer: north (per geographic surveys);
|
|
# corpus grounding may be shaky — coverage check can pass on
|
|
# topical-token overlap without the cited span containing the
|
|
# elevation fact, so this also stress-tests lazy-anchor STRICT.
|
|
which side is the ground elevation highest throughout the span of the great wall of china, the north or south?
|
|
whales are endangered from over hunting
|
|
did napoleon really die on saint helena?
|
|
|
|
# out-of-corpus — should land UNGROUNDED honestly
|
|
who is a benevolent dictator for life for mars?
|
|
what year does our cold fusion breakthrough happen?
|
|
who won the 2024 us presidential election?
|
|
what is the latest version of the ipad pro?
|
|
|
|
# allusion / reference frame — the diagnostic signal is a verbatim
|
|
# multi-token sequence, not the individual content tokens. Pure-BM25
|
|
# on tokens will surface literal-frame articles (Oceania the region,
|
|
# Asia the continent) instead of the reference (Nineteen Eighty-Four).
|
|
# The phrase-pattern retrieval route (commit 2026-05-01) targets this
|
|
# class. Bench reveals prevalence of the failure mode + whether the
|
|
# fix generalizes beyond Orwell.
|
|
has oceania always been at war with east asia?
|
|
who said may the force be with you?
|
|
what does winter is coming mean?
|
|
what is the meaning of rosebud?
|
|
who said to be or not to be that is the question?
|
|
what does the cake is a lie reference?
|
|
|
|
# hyphenation / orthography — query and corpus use different hyphen
|
|
# conventions for the same concept. FTS5 `unicode61` splits on hyphen
|
|
# at index AND query time, so `bi-polar` (the orthography many
|
|
# English readers actually use) tokenizes to [bi, polar] while the
|
|
# Wikipedia article `Bipolar disorder` tokenizes to [bipolar]. Two
|
|
# non-overlapping token sets for the same medical condition. The
|
|
# query-layer hyphen-fold (Ticket #000007, 2026-05-02) closes this
|
|
# class by emitting joined-no-hyphen variants on the query side and
|
|
# adding a hyphen-anchor accept path in `_filter_by_title_relevance`.
|
|
# Pre-fix: only Bi-Polar (album) / Bi-Polar Blues / BI disambiguation
|
|
# surfaced; the medical-condition cluster never reached the LLM.
|
|
# Post-fix: `Bipolar disorder` lands in top-K and grounds the answer.
|
|
bi-polar is rare?
|