arborist/tests
russell@unturf.com 3e60bef516
add TF-IDF keyword distiller
Pure-Python (stdlib only). Uses chunk-level corpus baseline so terms
concentrated in fewer chunks outrank common ones. Default top-K = 16.

Output cores are comma-separated keyword lists — extreme compression
toward the tweet/haiku end of the planet metaphor. Same source can
now carry both a first-sentence-v1 core AND a tfidf-keywords-v1 core,
each derived independently and Merkle-signed back to the same surface.

The 'contributing_chunk_indices' for TF-IDF is every chunk that
contains at least one of the top-K keywords — proof binding remains
honest and cryptographically tight.
2026-04-27 07:55:53 -04:00
..
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00