russell@unturf.com
8d6961fcc1
modified: .gitlab-ci.yml modified: bench/qa_questions.txt modified: bench/qa_sweep.py modified: bench/run.sh modified: docs/TICKETS.md modified: docs/_source/README.md modified: docs/_source/_ext/makefile_targets.py modified: docs/_source/api/cli.rst modified: docs/_source/api/distill.rst modified: docs/_source/api/mesh.rst modified: docs/_source/api/qa.rst modified: docs/_source/api/retrieval.rst modified: docs/_source/api/storage.rst modified: docs/_source/api/substrate.rst modified: docs/_source/concepts.rst modified: docs/_source/conf.py modified: docs/_source/cookbook.rst modified: docs/_source/index.rst modified: docs/_source/license.rst modified: docs/_source/quickstart.rst modified: docs/bench-maxing.md modified: docs/benchmarks.md modified: docs/cti-architecture.md modified: docs/diagrams/aborist-modules.dot modified: docs/diagrams/aborist-modules.svg modified: docs/diagrams/mesh-data-flow.dot modified: docs/diagrams/mesh-epoch-lifecycle.dot modified: docs/diagrams/mesh-epoch-lifecycle.svg modified: docs/diagrams/mesh-group-decisions.dot modified: docs/diagrams/mesh-group-decisions.svg modified: docs/diagrams/mesh-identity-stack.dot modified: docs/diagrams/mesh-secret-envelope.dot modified: docs/mesh.md modified: docs/qa-modes-bench.md modified: docs/seven-point-program.md modified: docs/tickets/ticket-000001-retrieval-keywords-audit-gap.md modified: docs/tickets/ticket-000002-reference-frame-polarity-contract.md modified: docs/tickets/ticket-000003-anchor-class-warrant.md modified: docs/tickets/ticket-000005-label-ladder-migration.md modified: docs/tickets/ticket-000006-bench-emergent-findings.md modified: docs/tickets/ticket-000007-query-layer-hyphen-fold.md modified: docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md modified: docs/tickets/ticket-000009-quantifier-preflight-dag-binding.md modified: docs/tickets/ticket-000010-metacognition-preflight-guard.md modified: docs/tickets/ticket-000011-soft-preflight-hint-sidecar.md modified: scripts/backfill_concepts.py modified: scripts/bench_emergent.py modified: tests/crawler/test_async_web_fetcher.py modified: tests/crawler/test_bridge.py modified: tests/crawler/test_web_fetch.py modified: tests/test_bench_qa_sweep.py modified: tests/test_burn.py modified: tests/test_burn_doc.py modified: tests/test_claim_lattice.py modified: tests/test_cli_render.py modified: tests/test_compress.py modified: tests/test_concepts.py modified: tests/test_dag.py modified: tests/test_directives.py modified: tests/test_distill.py modified: tests/test_distill_recursive.py modified: tests/test_evict.py modified: tests/test_frame.py modified: tests/test_grok_source.py modified: tests/test_html_source.py modified: tests/test_ingest.py modified: tests/test_inspect.py modified: tests/test_journal.py modified: tests/test_keys.py modified: tests/test_llm_context_base.py modified: tests/test_merkle.py modified: tests/test_mesh.py modified: tests/test_mesh_aead.py modified: tests/test_mesh_chain.py modified: tests/test_mesh_cli.py modified: tests/test_mesh_cli_pull.py modified: tests/test_mesh_wire.py modified: tests/test_mesh_wire_e2e.py modified: tests/test_metacognition.py modified: tests/test_migration_audit_mode.py modified: tests/test_providence_source.py modified: tests/test_qa.py modified: tests/test_qa_quality_live.py modified: tests/test_quantifier_caps.py modified: tests/test_quantifier_classifier.py modified: tests/test_quantifier_phase4.py modified: tests/test_quantifier_reminder.py modified: tests/test_query.py modified: tests/test_reclassify.py modified: tests/test_repair.py modified: tests/test_resume.py modified: tests/test_snapshot.py modified: tests/test_soft_preflight.py modified: tests/test_tfidf.py modified: tests/test_vcs_source.py modified: tests/test_verify.py modified: tests/test_verify_json.py modified: tests/test_versioned_ingest.py modified: tests/test_warrant.py modified: tests/test_wikipedia_old.py modified: tests/test_wikipedia_xml.py modified: tests/test_wikitext.py
143 lines
5.1 KiB
ReStructuredText
143 lines
5.1 KiB
ReStructuredText
Cookbook
|
||
========
|
||
|
||
Recipes for common workflows beyond the quickstart. Each starts from
|
||
a working arborist install (``make bootstrap`` already run) and a
|
||
populated shards directory under ``~/.arborist/shards/``.
|
||
|
||
Re-crawl a website to detect changes
|
||
-------------------------------------
|
||
|
||
After ``make crawl-ingest`` lands a site into a shard, the per-page
|
||
ETag and Last-Modified headers are kept in ``document_http_meta``. A
|
||
re-crawl can ask "did anything change?" without downloading bodies.
|
||
|
||
.. code-block:: sh
|
||
|
||
make crawl-ingest URL=https://example.com DEPTH=2 # initial crawl
|
||
# ...later...
|
||
make recrawl-check DOMAIN=example.com # conditional HEAD per page
|
||
|
||
Each URL is classified ``fresh`` (304), ``stale`` (200 with new body),
|
||
``gone`` (404/410), or ``unreachable``. One tiny round-trip per URL,
|
||
no body transfer when content is unchanged.
|
||
|
||
Falsify a wrong answer (audit-preserving)
|
||
------------------------------------------
|
||
|
||
The verifier called something STRICT but you know it's wrong. Mark
|
||
the record falsified — it stays in the DB so downstream consumers
|
||
that referenced it can still trace history.
|
||
|
||
.. code-block:: sh
|
||
|
||
make query Q="When did X happen?" # see the answer + cache_key
|
||
make inspect KEY=<cache_key> # diagnose unverified spans
|
||
make falsify KEY=<cache_key> REASON='wrong year — sources cited 1942 not 1944'
|
||
|
||
Future lookups skip records whose ``falsification_state != 'live'``.
|
||
A ``falsify`` audit event records the act; the chain stays intact.
|
||
|
||
Promote your own past answers into the corpus
|
||
----------------------------------------------
|
||
|
||
After enough STRICT answers accumulate, treat them as a derived
|
||
source. The ``providence`` ingest path promotes mature STRICT records
|
||
into the document corpus where retrieval can pick them up.
|
||
|
||
.. code-block:: sh
|
||
|
||
make ingest-self-providence KG_SECONDS=86400 # only records ≥1 day old
|
||
|
||
The ``KG_SECONDS`` (kindergarten window) prevents the system from
|
||
trusting freshly-cached answers as ground truth before they've had
|
||
time to fail. See :doc:`api/qa` for the ``ProvidenceSource`` impl.
|
||
|
||
Query across mixed corpora
|
||
--------------------------
|
||
|
||
Every shard under ``~/.arborist/shards/`` is queried automatically.
|
||
Mix Wikipedia, your Grok export, a crawled site, and your own git
|
||
repos in one query — retrieval ranks across all of them.
|
||
|
||
.. code-block:: sh
|
||
|
||
make ingest-cur-attached # Wikipedia
|
||
make ingest-grok-attached GROK_EXPORT=$HOME/Downloads/<uuid> # Grok chats
|
||
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # personal site
|
||
make ingest-git GIT_REPO=$HOME/git/myproject # source code
|
||
|
||
make query Q="how does my project handle authentication?"
|
||
|
||
Each shard contributes hits; the query path's title-relevance + body
|
||
coverage rerank lets neologisms in your private corpus outrank generic
|
||
Wikipedia matches.
|
||
|
||
Override the LLM endpoint
|
||
-------------------------
|
||
|
||
Default points at ``https://hermes.ai.unturf.com/v1`` (Hermes-3-8B,
|
||
no auth). Point at any OpenAI-compatible endpoint via env:
|
||
|
||
.. code-block:: sh
|
||
|
||
export ARBORIST_LLM_ENDPOINT="https://your-vllm.example.com/v1"
|
||
export ARBORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
|
||
export ARBORIST_LLM_API_KEY="..." # optional; many vLLM deploys are open
|
||
make query Q="..."
|
||
|
||
The model id folds into ``model_profile_hash`` (one of the 8 cache
|
||
key dimensions), so swapping models invalidates prior cache hits on
|
||
lookup — no risk of serving an answer one model produced under
|
||
another model's identity.
|
||
|
||
Verify shard integrity after a bulk operation
|
||
----------------------------------------------
|
||
|
||
Any state-changing op (mass falsify, hash bump, schema migration)
|
||
should be followed by:
|
||
|
||
.. code-block:: sh
|
||
|
||
make chain-check-shards # 0 chain breaks per shard = intact
|
||
make analyze-shards # compression spectrum + audit integrity
|
||
make verify-shards # round-trip Merkle proofs on a sample
|
||
|
||
Chain breaks are the loudest possible signal. Run these before
|
||
declaring an op successful.
|
||
|
||
Run the QA bench and read the results
|
||
--------------------------------------
|
||
|
||
The QA bench measures how often the verifier says STRICT vs HYBRID
|
||
vs UNGROUNDED across a fixed question set, per answer mode.
|
||
|
||
.. code-block:: sh
|
||
|
||
make bench-qa-smoke # ~30s, 5 questions × 3 modes
|
||
make bench-qa BENCH_QA_N=3 # full sweep, 3 samples each
|
||
|
||
Output lands in ``bench/qa_results/<utc-stamp>.{jsonl,md}``. The
|
||
markdown file has the summary table; the JSONL has every per-question
|
||
record for drill-down.
|
||
|
||
Resume an interrupted bench:
|
||
|
||
.. code-block:: sh
|
||
|
||
make bench-qa --resume bench/qa_results/<previous-utc-stamp>.jsonl
|
||
|
||
Same ``--seed`` is required for shuffled-task-order alignment.
|
||
|
||
Tune retrieval per-question
|
||
---------------------------
|
||
|
||
When a query returns the wrong sources, ``K=`` injects extra retrieval
|
||
keywords without changing what the LLM sees as the question:
|
||
|
||
.. code-block:: sh
|
||
|
||
make query Q="What did Orwell mean by always at war?" K="1984 Oceania Eastasia"
|
||
|
||
Provenance gap on this is tracked in
|
||
:doc:`api/qa` (``arborist.qa.query``).
|