russell@unturf.com
e561dfc599
Question-set churn (bench/qa_questions.txt): - narrow factoid: drop 'who founded apple computer?'; add 'a bridge between new london & groton?' (single-answer factoid with named-entity grounding via Gold Star Memorial Bridge) - broad descriptive: drop 'tell me about the apollo program' - entity list: drop 'list the founders of microsoft'; add 'name simpsons family members including pets?', 'list of obelisk in connecticut' - leading/adversarial: add 'whales are endangered from over hunting' (declarative-shape, tests verifier under premise-seeking phrasing) - out-of-corpus: rephrase Mars no-such-entity question to 'who is a benevolent dictator for life for mars?' — sharper adversarial shape since BDFL is a real OSS-governance term the model is tempted to project onto Mars - minor edits: 'GNU linux', 'C programming language', 'method man', 'cold fusion breakthrough' phrasing tests/test_qa_quality_live.py: rename test_mars_prime_minister_* to test_mars_benevolent_dictator_*; refresh refutation markers to match the new BDFL phrasing; expand to cover 'no one' / 'no person' negations the model uses on this shape. |
||
|---|---|---|
| .. | ||
| qa_questions.txt | ||
| qa_sweep.py | ||
| run.sh | ||