engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arborist/tests
History
russell@unturf.com 2af28e66af
bench: land #000023 + #000024 + #000025 (Phase 1a/1b — Dav1DPrometheus suite) 
Implements three coupled tickets in one push: complete the 5S battery,
align 5T to Dav1DPrometheus vocabulary + complete it, open 5F battery.
All runners deterministic, no LLM-as-judge anywhere.

#000023 — 5S Phase 1b (closed)
  run_syllogism + run_synthesis + run_semiotics replace stubs.
  30 deterministic fixtures each (90 total, all passing).
  Carrier metadata mandatory; unsupported carriers fail explicitly.
  Syllogism kernel handles categorical_transitivity, chain_3,
  invalid_converse, missing_premise. Order-agnostic over premise
  permutations.
  Synthesis uses content-token subset check (stopwords removed) —
  catches single-token entity swaps that lax overlap missed.
  Semiotics validates synonym-swap invariance via π* canonicalize +
  re-substitution. Hidden-channel work stays defensive only.

#000024 — 5T Phase 1b (closed)
  Vocabulary aligned: Transfer→transfer-learning, Truth→truthtables,
  Timing→time. Legacy transfer-v1.jsonl + run_transfer kept intact.
  run_transfer_learning, run_triangulation, run_truthtables,
  run_transitivity, run_time replace stubs.
  30 fixtures each × 5 = 150 total (155 with legacy transfer).
  Triangulation runs 4 strategies (substring, token_subset,
  token_overlap, entity_match) and gates on agreement threshold.
  Truthtables: deterministic propositional evaluator with
  recursive-descent parser; supports AND/OR/NOT/XOR/IMPL/IFF;
  capped at N=4 variables.
  Transitivity: typed-relation whitelist
  (implies, subset_of, ancestor_of, before, less_than). BFS path
  walk; mixed/unknown relations fail by construction.
  Time: synthetic memory-snapshot chains test preservation,
  stale-marking, current_root tracking. Phase 1b.2 will read real
  memory_records.

#000025 — 5F Phase 1a (in progress; Phase 1b expansion still open)
  New bench/batteries/b_5f.py with five sub-batteries:
  - Function (shape_match / pointer_set_match / threshold_on_metric)
  - Finetuning (parent→child SelfModel improvement check)
  - Falsification (planted-error detection rate, verifier_method_root pinned)
  - Formulate (structural lattice match: claim count + sorted-approx
    text + exact pointer-ID-set; not exact-string)
  - Feedback Loop (operation/observation chain → expected_delta in
    aggregated observation feed)
  10 deterministic seed fixtures per sub-battery (50 total).
  feedback_efficiency / adaptation_efficiency hooks stub the capital-
  ledger integration for v8 fork-choice.

Cross-cutting:
  - bench/batteries/base.py: PHASE_1_CARRIERS whitelist +
    validate_carrier helper. Backward-compatible with Phase 1a
    fixtures that lack carrier field (defaults to "text").
  - runner.py registers all 16 sub-batteries in _DEFAULT_FIXTURES
    so `--all` runs the entire Dav1DPrometheus suite.
  - Makefile: bench-5s / bench-5t / bench-5f / bench-5s5t5f
    targets. bench-5t-legacy preserves Phase 1a access.
  - tests/test_bench_batteries.py: 29 tests (was 17). Phase 1a
    digest stability checks, sub-battery smoke tests, carrier
    rejection tests, transitivity-whitelist test, truthtables
    N>4 cap test.

Bench summary across the full suite:
  5s syntax/semantics/syllogism/synthesis/semiotics  108/108 pass
  5t transfer/transfer-learning/triangulation/truthtables/transitivity/time  154/154 pass
  5f function/finetuning/falsification/formulate/feedback-loop  50/50 pass
  TOTAL: 312 fixtures across 16 sub-batteries — 100% pass.

Full test suite: 1103 passed, 36 skipped.

Source: Legally Unprecedented Dav1DPrometheus (BasementAGI host).
Honoring his framework. The complete state-space synthesis (SQD +
v7 + 5S/5T/5F) is now executable infrastructure, not metaphor.
2026-05-07 20:14:44 -04:00
..
crawler aborist/arborist 2026-05-07 09:31:49 -04:00
fixtures verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_bench_batteries.py bench: land #000023 + #000024 + #000025 (Phase 1a/1b — Dav1DPrometheus suite) 2026-05-07 20:14:44 -04:00
test_bench_qa_sweep.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_burn.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_burn_doc.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_capital.py capital: land ticket #000020 (8-capital-form cost ledger) 2026-05-07 16:41:48 -04:00
test_claim_lattice.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_cli_render.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_compress.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_concepts.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_dag.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_directives.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_distill.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_distill_recursive.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_evict.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_frame.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_grok_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_html_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_ingest.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_inspect.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_journal.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_keys.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_llm_context_base.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_loss_report.py loss_report: land ticket #000022 (adapter LossReport sidecar) 2026-05-07 17:58:50 -04:00
test_memory_root.py memory: land ticket #000017 (memory-root lifelong learning summary) 2026-05-07 16:46:41 -04:00
test_merkle.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_aead.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_chain.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_cli.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_cli_pull.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_wire.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_mesh_wire_e2e.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_metacognition.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_migration_audit_mode.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_pi_star.py pi_star: land ticket #000015 (π* domain library + composition algebra) 2026-05-07 16:51:33 -04:00
test_providence_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_qa.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_qa_quality_live.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_caps.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_classifier.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_phase4.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_quantifier_reminder.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_query.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_reclassify.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_repair.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_resume.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_selfmodel.py selfmodel: land ticket #000014 (identity record + falsification) 2026-05-07 16:36:34 -04:00
test_snapshot.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_soft_preflight.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_tfidf.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_vcs_source.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_verify.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_verify_json.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_versioned_ingest.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_warrant.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_wikipedia_old.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_wikipedia_xml.py aborist/arborist 2026-05-07 09:31:49 -04:00
test_wikitext.py aborist/arborist 2026-05-07 09:31:49 -04:00