russell@unturf.com
8d6961fcc1
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py
112 lines
4.4 KiB
ReStructuredText
112 lines
4.4 KiB
ReStructuredText
Quickstart
|
||
==========
|
||
|
||
Two end-to-end paths. Pick whichever corpus you want first; both share
|
||
the same query, verify, falsify, and inspect surfaces.
|
||
|
||
Install
|
||
-------
|
||
|
||
Arborist needs Python 3.10+, GNU make, ``curl``, and ``bzip2``. SQLite
|
||
3.35+ ships with CPython.
|
||
|
||
.. code-block:: sh
|
||
|
||
git clone https://git.unturf.com/engineering/unturf/arborist.git
|
||
cd arborist
|
||
make bootstrap # one-time: venv + dev extras
|
||
|
||
``make bootstrap`` creates ``.venv/``, installs the package in editable
|
||
mode with ``[dev,html]`` extras, and exposes ``arborist`` at
|
||
``.venv/bin/arborist``. No system-wide install. Re-running is a no-op
|
||
if the venv is up to date.
|
||
|
||
After bootstrap, every workflow lives behind a ``make`` target. Run
|
||
``make help`` (or see the auto-generated :doc:`api/makefile` reference)
|
||
to list them.
|
||
|
||
Path A — Wikipedia 2003 (canonical bootstrap dataset)
|
||
------------------------------------------------------
|
||
|
||
.. code-block:: sh
|
||
|
||
make fetch-cur # download 2003-05-16 snapshot (~82 MB)
|
||
make ingest-cur-attached # ~3 min: 128k articles, 4 parallel shards
|
||
make distill-shards-parallel # surface → core (first-sentence)
|
||
make distill-shards-tfidf-parallel # core → keyword sets for retrieval
|
||
make query Q="What is anarcho-capitalism?"
|
||
|
||
A Hermes-3 inference runs against the local corpus, picks 4–8 source
|
||
articles by Merkle root, and returns an answer plus a verifier label
|
||
that names what the lexical verifier could confirm. The four-rung
|
||
ladder for claim-lattice modes is ``POINTER-LINKED`` →
|
||
``ANCHOR-WARRANTED`` → ``EVIDENCE-WARRANTED`` → ``UNGROUNDED``
|
||
(with ``-PARTIAL`` suffix on HYBRID). Repeat the same question and a
|
||
cache hit replays in ~100 ms.
|
||
|
||
Path B — Crawl any live website and query it
|
||
---------------------------------------------
|
||
|
||
.. code-block:: sh
|
||
|
||
make bootstrap-crawler # one-time: install [crawler] extras
|
||
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # BFS + ingest
|
||
make query Q="who is Russell Ballestrini?" # cross-shard; picks up new shard automatically
|
||
|
||
The crawl shard is named after the seed hostname
|
||
(``crawl_russell_ballestrini_net.db``) under ``~/.arborist/shards/``.
|
||
``FAST=1`` enables aggressive crawling for your own sites; ``MAX=N``
|
||
caps discovery; ``DEPTH=N`` bounds BFS. Robots ``Disallow`` is always
|
||
honored. After ingest, ``make recrawl-check DOMAIN=...`` does a
|
||
conditional-HEAD freshness probe per page.
|
||
|
||
After the answer
|
||
----------------
|
||
|
||
.. code-block:: sh
|
||
|
||
make inspect KEY=<cache_key> # sidecar: classify each unverified span
|
||
make falsify KEY=<cache_key> REASON='…' # mark wrong, keep history
|
||
make burn KEY=<cache_key> REASON='…' # delete (kindergarten only — refuses if children exist)
|
||
|
||
The query path
|
||
--------------
|
||
|
||
.. figure:: diagrams/query-pipeline.svg
|
||
:alt: Query pipeline
|
||
:width: 100%
|
||
|
||
Every query runs through the same pipeline:
|
||
|
||
1. **Search** — FTS5 (body) + SQL ``LIKE`` (title) + ``JOIN`` over
|
||
derivations (TF-IDF core keywords) across every shard.
|
||
2. **Concept overlay** — per-shard ``concept_relations`` SQLite table
|
||
widens retrieval via synonyms, narrows via rivalries unless the
|
||
query uses comparative phrasing.
|
||
3. **Context assembly** — top-K sources concatenated up to a 60 KB
|
||
budget. Wikitext stripped to plain prose.
|
||
4. **LLM** — Hermes-3 with strict attribution rules.
|
||
5. **Verifier** — every claim runs through a layered lexical check;
|
||
result rolls up into the v9.8 trichotomy
|
||
(``audit_mode ∈ STRICT / HYBRID / UNGROUNDED``) at the schema layer
|
||
AND a four-rung display ladder at render time.
|
||
6. **Cache** — the v9.8 8-dim ``cache_key`` keys the answer in the
|
||
shard. Cache hits replay in ~100 ms.
|
||
|
||
LLM endpoint defaults to ``https://hermes.ai.unturf.com/v1`` (Hermes-3
|
||
Llama-3.1-8B, 82K context, no auth). Override:
|
||
|
||
.. code-block:: sh
|
||
|
||
export ARBORIST_LLM_ENDPOINT="https://your-vllm.example/v1"
|
||
export ARBORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
|
||
export ARBORIST_LLM_API_KEY="..."
|
||
|
||
Where to next
|
||
-------------
|
||
|
||
* :doc:`api/makefile` — every ``make`` target with a one-line description
|
||
* :doc:`api/cli` — direct ``arborist`` CLI reference
|
||
* :doc:`api/qa` — Q&A pipeline internals (verifier, evidence, DAG)
|
||
* :doc:`api/substrate` — Merkle tree + document primitives
|
||
* :doc:`license` — full AGPL + Permacomputer Preamble
|