ticket #000031 follow-ups B-1 + B-2: alias attribution + source-side title author

B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.

  - New Citation.via_citation_alias field (default False, preserves
    parse-from-source-ref path).
  - warrant_status sets via_citation_alias=True on substitute
    Citations from #000041 lookup_citation_aliases.
  - resolve_chunks reads citation.via_citation_alias as a "floor" for
    via_alias on every match it produces (Pass 1 hits inherit it
    too, not just Pass 2 term-alias hits). Audit-honest: matches
    from a substitute Citation are alias-driven regardless of which
    cascade pass found the chunk.

  Live re-resolve under the new attribution: 18 direct + 74 +alias
  (was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
  records resolving on the literally-cited Hilbert textbook. All 74
  records resolved via citation-alias substitution now carry
  process_id="warrant-resolver-v1+alias" in derivations.

  3 new unit tests (test_warrant_resolver.py): default-False on
  parsed Citations, explicit-True construction works, resolve_chunks
  propagates the floor onto every ResolutionMatch.

B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.

  - HtmlPageSource accepts default_author kwarg; appends ', by
    <author>' to ingested document titles when the <title> tag
    doesn't already include the surname.
  - TextbookTexSource accepts default_author kwarg; appends ' by
    <author>' to titles when the LaTeX has no \author{} macro AND
    no PG-style 'Author:' boilerplate.
  - _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
    kwarg; same append logic. ingest_crawled() and arborist crawl
    --ingest plumb it through.
  - arborist ingest --author + arborist crawl --author CLI flags.
  - bench/scripts/textbooks_manifest.py:cmd_lookup emits the
    manifest's `author` field as a 7th tab column.
  - make textbook target reads the author column and threads
    --author into both crawl-ingest and shallow-ingest paths.

  Idempotency preserved — surname-already-in-title detection prevents
  double-stamping on re-ingest. Shards previously SQL-backfilled
  (Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
  Grinstead-Snell) keep their existing titles; new ingests pick up
  the author signal at source time.

  Live smoke: arborist ingest --source html --author "Bertrand
  Russell" against PG #41654 yields title "Introduction to
  Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
  with no SQL UPDATE needed.

Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.

Makefile

@@ -766,20 +766,24 @@ textbook: bootstrap-crawler ## ingest one textbook by id: make textbook ID=bogar
 	max=$$(echo "$$row" | cut -f4); \
 	license=$$(echo "$$row" | cut -f5); \
 	domain=$$(echo "$$row" | cut -f6); \
+	author=$$(echo "$$row" | cut -f7); \
 	mkdir -p $(CRAWL_SHARDS_DIR); \
 	shard="$(CRAWL_SHARDS_DIR)/textbook_$${id}.db"; \
+	author_flag=""; \
+	if [ -n "$$author" ]; then author_flag="--author $$author"; fi; \
 	echo ">> $$id  ($$license, $$domain)"; \
 	echo "   crawl: $$url  depth=$$depth  max=$$max"; \
 	echo "   shard: $$shard"; \
+	if [ -n "$$author" ]; then echo "   author: $$author"; fi; \
 	if [ -n "$$url" ]; then \
-	  $(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest > /tmp/textbook-$${id}.log 2>&1 && \
+	  $(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest $$author_flag > /tmp/textbook-$${id}.log 2>&1 && \
 	  docs=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM documents\").fetchone()[0])"); \
 	  chunks=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM chunks\").fetchone()[0])"); \
 	  echo "   landed: $$docs docs / $$chunks chunks"; \
 	else \
 	  echo "   no crawl_url for $$id; using shallow ingest path"; \
 	  echo "$$id" | $(PY) -c 'import sys, json, os; sys.path.insert(0, "."); ids = sys.stdin.read().split(); manifest = "$(TEXTBOOK_MANIFEST)"; [print(u) for line in open(manifest) for e in [json.loads(line)] if "_meta" not in e and e.get("id") in ids for u in e.get("urls", [])]' > /tmp/textbook-$${id}-urls.txt; \
-	  $(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt; \
+	  $(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt $$author_flag; \
 	  rm -f /tmp/textbook-$${id}-urls.txt; \
 	fi
 

arborist/cli.py

@@ -56,7 +56,11 @@ def _cmd_ingest(args: argparse.Namespace) -> int:
         if not urls:
             print("html source needs --url or --urls-from", file=sys.stderr)
             return 2
-        src = HtmlPageSource(urls, respect_robots=not args.no_robots)
+        src = HtmlPageSource(
+            urls,
+            respect_robots=not args.no_robots,
+            default_author=getattr(args, "author", None),
+        )
     elif args.source in ("grok_export", "grok_media"):
         if not args.path:
             print(f"--path is required for {args.source}", file=sys.stderr)
@@ -87,7 +91,10 @@ def _cmd_ingest(args: argparse.Namespace) -> int:
             return 2
         from arborist.sources import TextbookTexSource
 
-        src = TextbookTexSource(urls)
+        src = TextbookTexSource(
+            urls,
+            default_author=getattr(args, "author", None),
+        )
     elif args.source == "claim_pack":
         # Companion JSON bundles (axiom + theorem packs) — see ticket
         # #000029. --bundle is repeatable so axiom-bundle and theorem-bundle
@@ -4192,7 +4199,12 @@ def _cmd_crawl(args: argparse.Namespace) -> int:
     ingest_progress = Progress(prefix="ingest ", total_estimate=len(urls))
     conn = connect(args.db)
     try:
-        result = ingest_crawled(conn, urls, progress=ingest_progress)
+        result = ingest_crawled(
+            conn,
+            urls,
+            progress=ingest_progress,
+            default_author=getattr(args, "author", None),
+        )
     finally:
         conn.close()
     print(json.dumps(
@@ -4320,6 +4332,18 @@ def build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="do not consult robots.txt (use only for explicitly authorized sites)",
     )
+    ingest.add_argument(
+        "--author",
+        default=None,
+        help=(
+            "default author surname for the html / textbook_tex sources "
+            "(#000031 Phase 1 follow-up). Appended to document title as "
+            "'<title>, by <author>' so the warrant resolver's "
+            "_shard_matches_citation heuristic finds the surname in "
+            "the title haystack. Wikisource and PG HTML pages rarely "
+            "include author in <title>; the manifest carries it instead."
+        ),
+    )
     ingest.add_argument(
         "--chunker", default=None, help="chunker name (default: tok-512-v1)"
     )
@@ -5673,6 +5697,17 @@ def build_parser() -> argparse.ArgumentParser:
             "against domains where aggressive fetching is acceptable."
         ),
     )
+    crawl_cmd.add_argument(
+        "--author",
+        default=None,
+        help=(
+            "default author surname (#000031 Phase 1 follow-up). "
+            "Appended to ingested document titles when the HTML's "
+            "<title> doesn't already carry the surname. Threads into "
+            "_shard_matches_citation's title-haystack for warrant "
+            "resolution. Only relevant with --ingest."
+        ),
+    )
     crawl_cmd.set_defaults(func=_cmd_crawl)
 
     crawler_cmd = sub.add_parser(

arborist/qa/warrant_resolver.py

@@ -61,6 +61,14 @@ class Citation:
     the fallback. ``section`` carries chapter / section / equation
     refs when present (e.g., "§1.2.6 equation (13)") for downstream
     chunk-resolution heuristics.
+
+    ``via_citation_alias`` flags Citations that were appended to a
+    record's citation list via `#000041` lookup (rather than parsed
+    from the original ``source_reference``). When True, every
+    ResolutionMatch produced from this Citation inherits
+    ``via_alias=True`` so the derivations `process_id` correctly
+    reflects the alias substitution. Defaults False to preserve the
+    parse-from-source path.
     """
 
     title: str = ""
@@ -68,6 +76,7 @@ class Citation:
     year: str = ""
     section: str = ""
     raw: str = ""
+    via_citation_alias: bool = False
 
     def is_empty(self) -> bool:
         return not (self.title or self.authors)
@@ -689,6 +698,14 @@ def resolve_chunks(
         except Exception:
             aliased_queries = []
 
+    # Citation-alias substitutions (#000041) propagate via_alias to
+    # every match they produce — Pass 1 hits from a substitute Citation
+    # are still alias-driven, just not via term-alias query expansion
+    # (#000042). Both alias mechanisms collapse to the same
+    # via_alias=True downstream so process_id correctly attributes
+    # the derivation as `warrant-resolver-v1+alias`.
+    citation_alias_floor = bool(getattr(citation, "via_citation_alias", False))
+
     matches: list[ResolutionMatch] = []
     for shard in candidate_shards:
         if not _shard_matches_citation(str(shard), citation):
@@ -710,7 +727,7 @@ def resolve_chunks(
                             chunk_idx=idx,
                             score=-float(rank or 0.0),
                             snippet=snippet or "",
-                            via_alias=False,
+                            via_alias=citation_alias_floor,
                         )
                     )
                 hit = True
@@ -1105,6 +1122,7 @@ def warrant_status(
                             title=sub.substitute_title,
                             authors=sub.substitute_authors,
                             raw=sub.substitute_ref,
+                            via_citation_alias=True,
                         )
                     )
             except Exception:

arborist/sources/crawler/bridge.py

@@ -216,9 +216,16 @@ class _CrawledHtmlSource:
 
     source_type = "html"
 
-    def __init__(self, urls: Iterable[str], *, timeout: float = 30.0):
+    def __init__(
+        self,
+        urls: Iterable[str],
+        *,
+        timeout: float = 30.0,
+        default_author: str | None = None,
+    ):
         self.urls = list(urls)
         self.timeout = timeout
+        self.default_author = (default_author or "").strip()
         # Filled in during iter_documents — keyed by document_uri (request
         # URL pre-redirect) so the caller can map document_root → metadata
         # via the documents.document_uri column after ingest.
@@ -249,6 +256,25 @@ class _CrawledHtmlSource:
                 doc = parse_html(str(resp.url), resp.text, self.source_type)
                 if doc is None:
                     continue
+                # Phase 1 follow-up of #000031 — append manifest author
+                # to title when present and not already a substring.
+                if self.default_author:
+                    title = (doc.title or "").strip()
+                    title_l = title.lower()
+                    has_surname = False
+                    for tok in self.default_author.split():
+                        tok_l = tok.lower()
+                        if len(tok_l) >= 4 and tok_l in title_l:
+                            has_surname = True
+                            break
+                    if not has_surname:
+                        from dataclasses import replace
+
+                        new_title = (
+                            f"{title}, by {self.default_author}" if title
+                            else f"by {self.default_author}"
+                        )
+                        doc = replace(doc, title=new_title)
                 self.http_meta[doc.uri] = {
                     "etag": resp.headers.get("etag"),
                     "last_modified": resp.headers.get("last-modified"),
@@ -263,6 +289,7 @@ def ingest_crawled(
     *,
     timeout: float = 30.0,
     progress: Progress | None = None,
+    default_author: str | None = None,
 ) -> dict:
     """Fetch each URL, ingest into `conn`, record ETag + Last-Modified.
 
@@ -276,7 +303,7 @@ def ingest_crawled(
     Pass ``progress`` for stderr heartbeats while ingest runs (the
     underlying ingest_source supports it natively).
     """
-    src = _CrawledHtmlSource(urls, timeout=timeout)
+    src = _CrawledHtmlSource(urls, timeout=timeout, default_author=default_author)
     stats = ingest_source(conn, src, progress=progress)
     written: list[dict] = []
     if not src.http_meta:

arborist/sources/html_page.py

@@ -169,13 +169,24 @@ class HtmlPageSource(Source):
         loss_report_enabled: bool = True,
         loss_report_excerpts: bool = True,
         loss_report_max_excerpt_bytes: int = 200,
+        default_author: str | None = None,
     ):
+        """``default_author`` (Phase 1 follow-up of #000031) — when the
+        ingested HTML's ``<title>`` doesn't already include the author
+        surname, this string is appended (``"<title>, by <author>"``).
+        Surfaces the author signal into the warrant resolver's
+        ``_shard_matches_citation`` haystack without a per-shard SQL
+        UPDATE workaround. Wikisource and Project Gutenberg HTML pages
+        rarely include author in ``<title>`` — the manifest entry
+        carries it instead. Skipped when None or empty.
+        """
         self.urls = list(urls)
         self.respect_robots = respect_robots
         self.timeout = timeout
         self.loss_report_enabled = loss_report_enabled
         self.loss_report_excerpts = loss_report_excerpts
         self.loss_report_max_excerpt_bytes = loss_report_max_excerpt_bytes
+        self.default_author = (default_author or "").strip()
         self._robots_cache: dict[str, urllib.robotparser.RobotFileParser] = {}
 
     @classmethod
@@ -220,7 +231,7 @@ class HtmlPageSource(Source):
                     loss_collector=collector,
                 )
                 if doc is not None:
-                    yield doc
+                    yield self._with_author_appended(doc)
 
     def _allowed(self, client: "httpx.Client", url: str) -> bool:
         parsed = urllib.parse.urlparse(url)
@@ -239,3 +250,35 @@ class HtmlPageSource(Source):
                 rp.allow_all = True
             self._robots_cache[origin] = rp
         return rp.can_fetch(USER_AGENT, url)
+
+    def _with_author_appended(self, doc):
+        """Phase 1 follow-up of #000031 — when the source ingest
+        produces a Document whose title doesn't carry the author
+        surname, append ``", by <default_author>"`` so the resolver's
+        author-surname haystack check fires.
+
+        Suppress the append when ``default_author`` is empty OR when
+        the existing title already contains a surname token of length
+        >= 4 from the configured author (case-insensitive substring).
+        Idempotent — re-ingesting the same URL with the same config
+        produces the same Document content_root because content
+        doesn't change, only the title text appended.
+        """
+        if not self.default_author or not doc:
+            return doc
+        title = (doc.title or "").strip()
+        # If any surname-shaped token from default_author already
+        # appears in title (case-insensitive), don't append.
+        title_l = title.lower()
+        for tok in self.default_author.split():
+            tok_l = tok.lower()
+            if len(tok_l) >= 4 and tok_l in title_l:
+                return doc
+        new_title = (
+            f"{title}, by {self.default_author}" if title
+            else f"by {self.default_author}"
+        )
+        # Document is a frozen-ish dataclass; replace title.
+        from dataclasses import replace
+
+        return replace(doc, title=new_title)

arborist/sources/textbook_tex.py

@@ -264,11 +264,24 @@ class TextbookTexSource(Source):
 
     source_type = "textbook_tex"
 
-    def __init__(self, urls: list[str] | str, *, timeout: float = 60.0):
+    def __init__(
+        self,
+        urls: list[str] | str,
+        *,
+        timeout: float = 60.0,
+        default_author: str | None = None,
+    ):
+        """``default_author`` (Phase 1 follow-up of #000031) — fallback
+        when the LaTeX source has no ``\\author{}`` macro and no PG-
+        style ``Author:`` boilerplate (e.g., Peano's mdnahas/Peano_Book
+        TeX). Wired into the title via ``"<title> by <author>"`` for
+        the warrant resolver's author-surname haystack check. Skipped
+        when None or empty."""
         if isinstance(urls, str):
             urls = [urls]
         self.urls = list(urls)
         self.timeout = timeout
+        self.default_author = (default_author or "").strip()
 
     def iter_documents(self) -> Iterator[Document]:
         with httpx.Client(
@@ -288,7 +301,16 @@ class TextbookTexSource(Source):
                     # Stripped output too thin to be useful — likely
                     # an error page or a non-TeX response.
                     continue
-                title = _extract_title(tex) or url
+                title = _extract_title(tex)
+                if not title:
+                    # No \title{} or PG Title: line — fall back to URL,
+                    # but still append default_author when configured
+                    # so the haystack picks up the surname.
+                    title = url
+                if self.default_author and not _title_has_author(
+                    title, self.default_author
+                ):
+                    title = f"{title} by {self.default_author}"
                 yield Document(
                     uri=str(resp.url),
                     content=prose,
@@ -314,6 +336,20 @@ _PG_TITLE_RE = re.compile(r"^Title:\s*(.+?)\s*$", re.MULTILINE)
 _PG_AUTHOR_RE = re.compile(r"^Author:\s*(.+?)\s*$", re.MULTILINE)
 
 
+def _title_has_author(title: str, author: str) -> bool:
+    """True if any surname-shaped token (length >= 4) from ``author``
+    appears in ``title`` (case-insensitive substring). Used to decide
+    whether to append ``default_author`` — skip when the title
+    already carries the surname (avoids double-stamping).
+    """
+    title_l = (title or "").lower()
+    for tok in (author or "").split():
+        tok_l = tok.lower()
+        if len(tok_l) >= 4 and tok_l in title_l:
+            return True
+    return False
+
+
 def _extract_title(tex: str) -> str:
     """Pull the title — prefer ``\\title{...}`` if present (LaTeX
     convention), fall back to ``Title:`` line (Project Gutenberg

bench/scripts/textbooks_manifest.py

@@ -180,9 +180,12 @@ def cmd_tex_targets(stream: TextIO) -> int:
 
 def cmd_lookup(stream: TextIO) -> int:
     """Read entry id from sys.argv[2]; emit tab-separated
-    `id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain` for the matching
+    `id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain\\tauthor` for the
-    entry. Used by per-book make targets to look up one textbook's
+    matching entry. Used by per-book make targets to look up one
-    crawl parameters from the manifest without grep wizardry.
+    textbook's crawl parameters + author signal (#000031 Phase 1
+    follow-up — author flows into ``arborist crawl --author`` /
+    ``arborist ingest --author`` so document titles carry the
+    surname downstream resolvers need).
     """
     target_id = sys.argv[2] if len(sys.argv) > 2 else ""
     if not target_id:
@@ -199,9 +202,10 @@ def cmd_lookup(stream: TextIO) -> int:
         crawl_url = entry.get("crawl_url", "")
         depth = entry.get("crawl_depth", 2)
         max_pages = entry.get("crawl_max", 80)
+        author = entry.get("author", "")
         print(
             f"{entry.get('id')}\t{crawl_url}\t{depth}\t{max_pages}\t"
-            f"{entry.get('license')}\t{entry.get('domain')}"
+            f"{entry.get('license')}\t{entry.get('domain')}\t{author}"
         )
         return 0
     print(f"# id {target_id!r} not found in manifest", file=sys.stderr)

tests/test_warrant_resolver.py

@@ -216,3 +216,125 @@ def test_build_record_query_cascade_orders_correctly():
     assert queries[0] == '"line incidence"'
     # The content-token AND-join should appear in the cascade.
     assert any(" AND " in q for q in queries[1:])
+
+
+# --- B-1: via_citation_alias attribution -----------------------
+
+
+def test_citation_via_citation_alias_default_false():
+    """Citations parsed from the original source_reference start
+    with via_citation_alias=False — backward-compatible."""
+    from arborist.qa.warrant_resolver import Citation, parse_citation
+
+    cs = parse_citation("Foundations of Geometry by David Hilbert")
+    assert all(c.via_citation_alias is False for c in cs)
+
+
+def test_citation_explicit_via_citation_alias_flag():
+    """Construct a substitute Citation as if from #000041 lookup —
+    via_citation_alias=True. Used by warrant_resolve to flag chains
+    that came through a citation-alias substitution rather than the
+    parsed source_reference."""
+    from arborist.qa.warrant_resolver import Citation
+
+    sub = Citation(
+        title="Russell IMP",
+        authors=("Bertrand Russell",),
+        raw="Russell IMP by Bertrand Russell",
+        via_citation_alias=True,
+    )
+    assert sub.via_citation_alias is True
+
+
+def test_resolve_chunks_propagates_via_citation_alias_to_match(tmp_path, monkeypatch):
+    """When resolve_chunks is called with a Citation whose
+    via_citation_alias=True is set, every ResolutionMatch it produces
+    inherits via_alias=True so process_id correctly attributes the
+    derivation as `warrant-resolver-v1+alias`."""
+    import sqlite3
+    from pathlib import Path
+
+    from arborist.qa.warrant_resolver import (
+        Citation,
+        ResolutionMatch,
+        resolve_chunks,
+    )
+
+    # Build a tiny fake shard cluster: shards/000.db (empty) +
+    # crawl/textbook_test.db with one document + one chunk that the
+    # _shard_matches_citation heuristic can find by author surname.
+    shards_dir = tmp_path / "shards"
+    crawl_dir = tmp_path / "crawl"
+    shards_dir.mkdir()
+    crawl_dir.mkdir()
+
+    from arborist.store import SCHEMA_SQL
+
+    main_db = shards_dir / "000.db"
+    sqlite3.connect(str(main_db)).executescript(SCHEMA_SQL).close()
+
+    sub_db = crawl_dir / "textbook_test.db"
+    conn = sqlite3.connect(str(sub_db))
+    conn.executescript(SCHEMA_SQL)
+    conn.execute(
+        "INSERT INTO documents "
+        "(document_root, document_uri, source_type, kind, "
+        " compression_depth, title, chunking_version, "
+        " canonicalization_version, schema_version, ingest_ts) "
+        "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
+        ("a" * 64, "https://test/", "html", "surface", 0,
+         "Test Book by Bertrand Russell", "tok-512-v1",
+         "norm-v1", "v9.8.0", 0),
+    )
+    conn.execute(
+        "INSERT INTO chunks (chunk_id, document_root, idx, leaf_hash, content) "
+        "VALUES (1, ?, 0, ?, ?)",
+        ("a" * 64, "h" * 64, "philosophy mathematics test sample"),
+    )
+    conn.execute(
+        "INSERT INTO chunks_fts (rowid, content) VALUES (1, ?)",
+        ("philosophy mathematics test sample",),
+    )
+    conn.commit()
+    conn.close()
+
+    # Citation with via_citation_alias=True (mimics what #000041
+    # citation-alias lookup appends).
+    sub_cit = Citation(
+        title="Test Book",
+        authors=("Bertrand Russell",),
+        raw="Test Book by Bertrand Russell",
+        via_citation_alias=True,
+    )
+    matches = resolve_chunks(
+        sub_cit,
+        shards_dir,
+        theorem_name="Axiom of Test",
+        record_content="philosophy mathematics test sample axiom",
+        limit=3,
+    )
+    assert matches, "expected a match for the substitute citation"
+    assert all(
+        isinstance(m, ResolutionMatch) and m.via_alias is True
+        for m in matches
+    ), "every match from a via_citation_alias=True Citation must be flagged via_alias"
+
+    # Sanity check the inverse — Citation without the flag → matches
+    # are via_alias=False (the existing direct-cascade path).
+    parsed_cit = Citation(
+        title="Test Book",
+        authors=("Bertrand Russell",),
+        raw="Test Book by Bertrand Russell",
+    )
+    plain_matches = resolve_chunks(
+        parsed_cit,
+        shards_dir,
+        theorem_name="Axiom of Test",
+        record_content="philosophy mathematics test sample axiom",
+        limit=3,
+    )
+    assert plain_matches
+    assert all(m.via_alias is False for m in plain_matches), (
+        "Citation parsed from source_ref (no via_citation_alias) "
+        "must NOT inherit via_alias=True from the citation_alias_floor"
+    )