Cookbook
========

Recipes for common workflows beyond the quickstart. Each starts from
a working arborist install (``make bootstrap`` already run) and a
populated shards directory under ``~/.arborist/shards/``.

Re-crawl a website to detect changes
-------------------------------------

After ``make crawl-ingest`` lands a site into a shard, the per-page
ETag and Last-Modified headers are kept in ``document_http_meta``. A
re-crawl can ask "did anything change?" without downloading bodies.

.. code-block:: sh

   make crawl-ingest URL=https://example.com DEPTH=2  # initial crawl
   # ...later...
   make recrawl-check DOMAIN=example.com              # conditional HEAD per page

Each URL is classified ``fresh`` (304), ``stale`` (200 with new body),
``gone`` (404/410), or ``unreachable``. One tiny round-trip per URL,
no body transfer when content is unchanged.

Falsify a wrong answer (audit-preserving)
------------------------------------------

The verifier called something STRICT but you know it's wrong. Mark
the record falsified — it stays in the DB so downstream consumers
that referenced it can still trace history.

.. code-block:: sh

   make query Q="When did X happen?"             # see the answer + cache_key
   make inspect KEY=<cache_key>                  # diagnose unverified spans
   make falsify KEY=<cache_key> REASON='wrong year — sources cited 1942 not 1944'

Future lookups skip records whose ``falsification_state != 'live'``.
A ``falsify`` audit event records the act; the chain stays intact.

Promote your own past answers into the corpus
----------------------------------------------

After enough STRICT answers accumulate, treat them as a derived
source. The ``providence`` ingest path promotes mature STRICT records
into the document corpus where retrieval can pick them up.

.. code-block:: sh

   make ingest-self-providence KG_SECONDS=86400   # only records ≥1 day old

The ``KG_SECONDS`` (kindergarten window) prevents the system from
trusting freshly-cached answers as ground truth before they've had
time to fail. See :doc:`api/qa` for the ``ProvidenceSource`` impl.

Query across mixed corpora
--------------------------

Every shard under ``~/.arborist/shards/`` is queried automatically.
Mix Wikipedia, your Grok export, a crawled site, and your own git
repos in one query — retrieval ranks across all of them.

.. code-block:: sh

   make ingest-cur-attached                                          # Wikipedia
   make ingest-grok-attached GROK_EXPORT=$HOME/Downloads/<uuid>      # Grok chats
   make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2     # personal site
   make ingest-git GIT_REPO=$HOME/git/myproject                      # source code

   make query Q="how does my project handle authentication?"

Each shard contributes hits; the query path's title-relevance + body
coverage rerank lets neologisms in your private corpus outrank generic
Wikipedia matches.

Override the LLM endpoint
-------------------------

Default points at ``https://hermes.ai.unturf.com/v1`` (Hermes-3-8B,
no auth). Point at any OpenAI-compatible endpoint via env:

.. code-block:: sh

   export ARBORIST_LLM_ENDPOINT="https://your-vllm.example.com/v1"
   export ARBORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
   export ARBORIST_LLM_API_KEY="..."   # optional; many vLLM deploys are open
   make query Q="..."

The model id folds into ``model_profile_hash`` (one of the 8 cache
key dimensions), so swapping models invalidates prior cache hits on
lookup — no risk of serving an answer one model produced under
another model's identity.

Verify shard integrity after a bulk operation
----------------------------------------------

Any state-changing op (mass falsify, hash bump, schema migration)
should be followed by:

.. code-block:: sh

   make chain-check-shards          # 0 chain breaks per shard = intact
   make analyze-shards              # compression spectrum + audit integrity
   make verify-shards               # round-trip Merkle proofs on a sample

Chain breaks are the loudest possible signal. Run these before
declaring an op successful.

Run the QA bench and read the results
--------------------------------------

The QA bench measures how often the verifier says STRICT vs HYBRID
vs UNGROUNDED across a fixed question set, per answer mode.

.. code-block:: sh

   make bench-qa-smoke                          # ~30s, 5 questions × 3 modes
   make bench-qa BENCH_QA_N=3                   # full sweep, 3 samples each

Output lands in ``bench/qa_results/<utc-stamp>.{jsonl,md}``. The
markdown file has the summary table; the JSONL has every per-question
record for drill-down.

Resume an interrupted bench:

.. code-block:: sh

   make bench-qa --resume bench/qa_results/<previous-utc-stamp>.jsonl

Same ``--seed`` is required for shuffled-task-order alignment.

Tune retrieval per-question
---------------------------

When a query returns the wrong sources, ``K=`` injects extra retrieval
keywords without changing what the LLM sees as the question:

.. code-block:: sh

   make query Q="What did Orwell mean by always at war?" K="1984 Oceania Eastasia"

Provenance gap on this is tracked in
:doc:`api/qa` (``arborist.qa.query``).
