From 7fedea3f5fc72fb225e1e352953cc67c326602f5 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Fri, 22 May 2026 07:42:32 -0400 Subject: [PATCH] docs: index docs/crawler.md in CLAUDE.md docs index --- CLAUDE.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 25bb618..3746b96 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -592,6 +592,9 @@ Architecture / ongoing work: - `docs/cti-architecture.md` — CTI Clause Tree Intelligence. - `docs/mesh.md` — mesh wire format + group-key state machine. +- `docs/crawler.md` — web crawler: BFS discovery, robots/feed + handling, polite vs `--fast`, and the content-addressed + diagnostics (dedupe by `document_root`, orphan finding). - `docs/benchmarks.md` — orientation: harnesses, fixtures, signal floor, make targets, bench-row schema, addenda index. Read first when running a bench.