crawler: verbatim lift from agents.ai.unturf.com (off by default)

Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:

  core/async_web_fetcher.py  ->  aborist/sources/crawler/async_web_fetcher.py
  core/web_fetch.py          ->  aborist/sources/crawler/web_fetch.py

Two source-side changes during the lift:

1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
   `import aborist.sources.crawler` raises ImportError with the install
   hint instead of leaking AttributeErrors deep in user code.

2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
   `has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
   replaced with NotImplementedError stubs. Aborist has no chat
   surface — fetch intent is detected at the application layer. The
   two test classes that exercised these triggers are
   `@pytest.mark.skip`'d with the same rationale.

Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.

Off by default:

- `[crawler]` extras section in pyproject.toml carries the heavy
  deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.

Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.

Makefile

@@ -31,7 +31,7 @@ SEARCH_Q     ?= computer
         ingest-grok ingest-grok-media \
         ingest-self ingest-git ingest-hg \
         verify search stats test docs chain-check chain-check-shards \
-        falsify burn clean clean-db clean-data help
+        falsify burn bootstrap-crawler test-crawler clean clean-db clean-data help
 
 all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
 
@@ -309,8 +309,17 @@ search: bootstrap ## keyword search; override SEARCH_Q (or pass Q=...)
 stats: bootstrap ## counts: documents, chunks, edges, audit chain
 	$(ABORIST) --db $(DB) stats
 
-test: bootstrap ## run pytest suite
+test: bootstrap ## run pytest suite (excludes opt-in crawler tests)
-	$(VENV)/bin/pytest -q
+	$(VENV)/bin/pytest -q --ignore=tests/crawler
+
+# Crawler tests are off-by-default — they hit the network in many cases
+# and require the heavy [crawler] extras (aiohttp, bs4, lxml, etc.).
+# Bootstrap installs the extras into the existing venv idempotently.
+bootstrap-crawler: bootstrap ## install [crawler] extras into the venv
+	$(PIP) install -e '.[crawler]'
+
+test-crawler: bootstrap-crawler ## run only the lifted crawler tests
+	$(VENV)/bin/pytest -q tests/crawler
 
 DOT_SRCS := $(wildcard docs/diagrams/*.dot)
 DOT_PNGS := $(DOT_SRCS:.dot=.png)

aborist/sources/crawler/__init__.py

@@ -0,0 +1,88 @@
+"""Crawler — ethical async web fetching with robots.txt compliance.
+
+Verbatim lift from ``~/git/agents.ai.unturf.com/core/`` (rev as of 2026-04-28):
+
+    core/async_web_fetcher.py  ->  aborist/sources/crawler/async_web_fetcher.py
+    core/web_fetch.py          ->  aborist/sources/crawler/web_fetch.py
+
+Two source-side adaptations during the lift:
+
+1. Chat-bot fetch triggers (``has_fresh_fetch_trigger`` /
+   ``has_web_fetch_trigger`` from ``core.keywords``) were dropped. Aborist
+   has no chat surface — fetch intent is detected at the application
+   layer, not from message content.
+
+2. ``web_cache_manager.py`` (SQLAlchemy-backed page cache) was NOT
+   lifted. Aborist has its own content-addressed store; cache through
+   that path instead of carrying SQLAlchemy as a dep.
+
+**Off by default.** Heavy dependencies (aiohttp, beautifulsoup4, lxml,
+html5lib, html2text, miniuri, feedparser, Pillow, cairosvg, pypdf) ship
+as the ``[crawler]`` extras. Importing this module without the extras
+raises ``ImportError`` with the install hint. The default test suite
+(``make test``) does not exercise the crawler; ``make test-crawler``
+runs the lifted tests with the extras installed.
+
+Public surface — same as the agents repo:
+
+    from aborist.sources.crawler import AsyncWebFetcher, CrawlMode
+    from aborist.sources.crawler import (
+        URIContentType,
+        extract_all_uris_from_content,
+        extract_url_from_content,
+        build_sources_footer,
+        fetch_and_cache,
+    )
+
+Adopters should pass ``user_agent`` reflecting their deployment and
+honor ``robots.txt`` — that's what the underlying parser already does
+when given a URL.
+"""
+
+from aborist.sources.crawler.async_web_fetcher import (
+    AsyncWebFetcher,
+    CrawlMode,
+    FeedItem,
+    MediaItem,
+    MediaMetadata,
+    discover_feeds,
+    extract_media_from_html,
+    fetch_robots_txt,
+    parse_atom,
+    parse_feed,
+    parse_rss,
+    parse_sitemap,
+)
+from aborist.sources.crawler.web_fetch import (
+    URIContentType,
+    build_sources_footer,
+    detect_uri_content_type,
+    extract_all_uris_from_content,
+    extract_domain,
+    extract_url_from_content,
+    fetch_and_cache,
+    format_progress_message,
+)
+
+__all__ = [
+    "AsyncWebFetcher",
+    "CrawlMode",
+    "FeedItem",
+    "MediaItem",
+    "MediaMetadata",
+    "URIContentType",
+    "build_sources_footer",
+    "detect_uri_content_type",
+    "discover_feeds",
+    "extract_all_uris_from_content",
+    "extract_domain",
+    "extract_media_from_html",
+    "extract_url_from_content",
+    "fetch_and_cache",
+    "fetch_robots_txt",
+    "format_progress_message",
+    "parse_atom",
+    "parse_feed",
+    "parse_rss",
+    "parse_sitemap",
+]

aborist/sources/crawler/web_fetch.py

@@ -0,0 +1,520 @@
+"""
+Web Fetch Module - Platform-agnostic web content fetching.
+
+This module provides URL extraction, web content fetching, and source citation
+building without platform-specific dependencies.
+"""
+
+import re
+import logging
+from enum import Enum
+from typing import Optional, Dict, List, Callable, Awaitable
+
+# Heavy crawler dependencies are optional. Install via:
+#     pip install 'aborist[crawler]'
+try:
+    from miniuri import Uri
+except ImportError as e:  # pragma: no cover
+    raise ImportError(
+        "crawler module requires extras: pip install 'aborist[crawler]'"
+    ) from e
+
+
+# `core.keywords` lived in agents.ai.unturf.com and detected fetch-intent
+# from chat-bot messages. Aborist has no chat surface, so the verbatim
+# lift drops those triggers. Callers asking "is this content a fetch
+# request?" should adapt at the application layer instead.
+def has_fresh_fetch_trigger(content: str) -> bool:  # pragma: no cover
+    raise NotImplementedError(
+        "chat-bot fetch triggers were dropped during the aborist lift; "
+        "callers should detect fetch intent at the application layer"
+    )
+
+
+def has_web_fetch_trigger(content: str) -> bool:  # pragma: no cover
+    raise NotImplementedError(
+        "chat-bot fetch triggers were dropped during the aborist lift; "
+        "callers should detect fetch intent at the application layer"
+    )
+
+
+logger = logging.getLogger(__name__)
+
+
+class URIContentType(Enum):
+    """Content type categories for URIs."""
+    PAGE = "page"          # HTML, text pages - fetch and extract
+    PDF = "pdf"            # PDF documents - fetch and extract
+    IMAGE = "image"        # Images (png, jpg, gif, webp) - use vision
+    SVG = "svg"            # SVG - convert to PNG then vision
+    VIDEO = "video"        # Video - skip or extract metadata
+    AUDIO = "audio"        # Audio - skip or transcribe
+    UNKNOWN = "unknown"    # Unknown - try fetch, fallback to vision
+
+
+# Content-type mapping
+CONTENT_TYPE_MAP = {
+    # Pages
+    'text/html': URIContentType.PAGE,
+    'application/xhtml+xml': URIContentType.PAGE,
+    'text/plain': URIContentType.PAGE,
+    'text/markdown': URIContentType.PAGE,
+    'application/json': URIContentType.PAGE,
+    'application/xml': URIContentType.PAGE,
+    'text/xml': URIContentType.PAGE,
+    # PDF
+    'application/pdf': URIContentType.PDF,
+    # Images
+    'image/png': URIContentType.IMAGE,
+    'image/jpeg': URIContentType.IMAGE,
+    'image/gif': URIContentType.IMAGE,
+    'image/webp': URIContentType.IMAGE,
+    'image/bmp': URIContentType.IMAGE,
+    'image/tiff': URIContentType.IMAGE,
+    # SVG (special handling)
+    'image/svg+xml': URIContentType.SVG,
+    # Video
+    'video/mp4': URIContentType.VIDEO,
+    'video/webm': URIContentType.VIDEO,
+    'video/ogg': URIContentType.VIDEO,
+    # Audio
+    'audio/mpeg': URIContentType.AUDIO,
+    'audio/ogg': URIContentType.AUDIO,
+    'audio/wav': URIContentType.AUDIO,
+}
+
+
+async def detect_uri_content_type(url: str, session=None) -> URIContentType:
+    """
+    Detect content type of URI using HEAD request.
+
+    Args:
+        url: URI to check
+        session: Optional aiohttp session to reuse
+
+    Returns:
+        URIContentType enum value
+    """
+    import aiohttp
+
+    try:
+        uri = Uri(url)
+        path = uri.path or ''
+
+        # Make HEAD request
+        close_session = False
+        if session is None:
+            session = aiohttp.ClientSession()
+            close_session = True
+
+        try:
+            async with session.head(
+                url,
+                allow_redirects=True,
+                timeout=aiohttp.ClientTimeout(total=10),
+                headers={'User-Agent': 'uncloseai.com/1.42 (ethical web crawler; +https://uncloseai.com)'}
+            ) as response:
+                content_type = response.headers.get('Content-Type', '')
+                # Strip charset etc: "text/html; charset=utf-8" -> "text/html"
+                mime_type = content_type.split(';')[0].strip().lower()
+
+                logger.info(f"HEAD {url} -> {response.status}, Content-Type: {mime_type}")
+
+                if mime_type in CONTENT_TYPE_MAP:
+                    return CONTENT_TYPE_MAP[mime_type]
+
+                # Fallback heuristics based on extension
+                if path.endswith('.pdf'):
+                    return URIContentType.PDF
+                elif path.endswith(('.png', '.jpg', '.jpeg', '.gif', '.webp')):
+                    return URIContentType.IMAGE
+                elif path.endswith('.svg'):
+                    return URIContentType.SVG
+
+                # Default to page for unknown
+                return URIContentType.PAGE
+
+        finally:
+            if close_session:
+                await session.close()
+
+    except Exception as e:
+        logger.warning(f"Failed to detect content type for {url}: {e}")
+        return URIContentType.UNKNOWN
+
+
+def extract_all_uris_from_content(content: str) -> List[str]:
+    """
+    Extract ALL URIs from message content using regex.
+
+    This function bypasses LLM tool calling to prevent URI hallucination.
+    It extracts all URIs with protocol first, then any domain-like patterns.
+    URIs inside code blocks are excluded.
+
+    Args:
+        content: Message content to extract URIs from
+
+    Returns:
+        List of extracted URI strings (may be empty)
+
+    Examples:
+        >>> extract_all_uris_from_content("Check https://a.com and https://b.com/file.css")
+        ['https://a.com', 'https://b.com/file.css']
+        >>> extract_all_uris_from_content("```python\\nurl = 'https://ignore.me'\\n```\\nVisit https://real.com")
+        ['https://real.com']
+    """
+    # Strip Matrix mentions first (e.g., @user:matrix.org) to avoid matching them as URIs
+    content_cleaned = re.sub(r'@[\w.-]+:[\w.-]+', '', content)
+
+    # Remove code blocks (``` ... ```) to avoid extracting URIs from code
+    content_cleaned = re.sub(r'```[\s\S]*?```', '', content_cleaned)
+    # Also remove inline code (`...`)
+    content_cleaned = re.sub(r'`[^`]+`', '', content_cleaned)
+
+    uris = []
+
+    # Find all URIs with protocol
+    for match in re.finditer(r'https?://[^\s<>]+', content_cleaned):
+        uri = match.group(0)
+        if uri not in uris:
+            uris.append(uri)
+            logger.info(f"Extracted URI with protocol: {uri}")
+
+    # Find URIs without protocol (e.g., "example.com", "media.unturf.com/test")
+    # Exclude common file extensions that look like TLDs
+    file_extensions = {'html', 'htm', 'css', 'js', 'jsx', 'tsx', 'ts', 'json', 'xml', 'svg',
+                       'png', 'jpg', 'jpeg', 'gif', 'webp', 'ico', 'pdf', 'txt', 'md',
+                       'py', 'java', 'cpp', 'hpp', 'go', 'rs', 'rb', 'php', 'sh', 'bash',
+                       'yaml', 'yml', 'toml', 'ini', 'conf', 'log', 'csv', 'sql'}
+
+    for match in re.finditer(r'(?:www\.)?(?:[a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(?:/[^\s<>]*)?', content_cleaned):
+        matched = match.group(0)
+
+        # Check if this is followed by :number (error line reference like "file.html:98:35")
+        end_pos = match.end()
+        if end_pos < len(content_cleaned) and content_cleaned[end_pos] == ':':
+            # Check if followed by digits
+            remaining = content_cleaned[end_pos+1:]
+            if remaining and remaining[0].isdigit():
+                logger.debug(f"Skipping error reference pattern: {matched}")
+                continue
+
+        # Extract the TLD (last part after final dot, before any path)
+        domain_part = matched.split('/')[0]  # Get just the domain
+        tld = domain_part.split('.')[-1].lower()
+
+        # Skip if it looks like a filename (TLD is a file extension)
+        if tld in file_extensions:
+            logger.debug(f"Skipping filename-like pattern: {matched}")
+            continue
+
+        uri = f"https://{matched}"
+        # Check it wasn't already captured with protocol
+        if uri not in uris and not any(uri in existing for existing in uris):
+            uris.append(uri)
+            logger.info(f"Extracted URI without protocol, added https://: {uri}")
+
+    if not uris:
+        logger.debug("No URIs found in content")
+
+    return uris
+
+
+def extract_url_from_content(content: str) -> Optional[str]:
+    """
+    Extract first URL from message content using regex.
+
+    This function bypasses LLM tool calling to prevent URL hallucination.
+    It first tries to match URLs with protocol (http:// or https://),
+    then falls back to matching domain-like patterns and prepending https://.
+
+    Args:
+        content: Message content to extract URL from
+
+    Returns:
+        Extracted URL string or None if no URL found
+
+    Examples:
+        >>> extract_url_from_content("Check out https://example.com")
+        'https://example.com'
+        >>> extract_url_from_content("Visit example.com")
+        'https://example.com'
+        >>> extract_url_from_content("What about media.unturf.com/test")
+        'https://media.unturf.com/test'
+    """
+    uris = extract_all_uris_from_content(content)
+    return uris[0] if uris else None
+
+
+def detect_fresh_fetch_intent(content: str) -> bool:
+    """
+    Detect if user wants to bypass cache and fetch fresh content.
+
+    Uses centralized multilingual keywords (fresh, refresh, reload, update, etc.)
+
+    Args:
+        content: Message content to analyze
+
+    Returns:
+        True if fresh fetch is requested, False otherwise
+    """
+    if has_fresh_fetch_trigger(content):
+        logger.info("Fresh fetch intent detected via centralized keywords")
+        return True
+    return False
+
+
+def build_sources_footer(source_urls: List[Dict[str, str]], format: str = "markdown") -> str:
+    """
+    Build sources footer from source URL metadata.
+
+    Args:
+        source_urls: List of dicts with 'url' and 'title' keys
+        format: Output format - "markdown" (Discord), "plain" (IRC/Matrix), or "html"
+
+    Returns:
+        Formatted sources section
+
+    Example:
+        >>> sources = [{'url': 'https://example.com', 'title': 'Example Page'}]
+        >>> build_sources_footer(sources)
+        '**Sources:**\\n- **Example Page**: <https://example.com>\\n'
+        >>> build_sources_footer(sources, format="plain")
+        'Sources: Example Page (https://example.com)'
+    """
+    if not source_urls:
+        return ""
+
+    if format == "plain":
+        # Single line, no markdown, no emoji - for IRC and Matrix
+        parts = []
+        for source in source_urls:
+            title = source.get('title', 'Untitled')
+            url = source.get('url', '')
+            if url:
+                parts.append(f"{title} ({url})")
+        return "Sources: " + ", ".join(parts) if parts else ""
+    else:
+        # Markdown format - for Discord
+        footer = "**Sources:**\n"
+        for source in source_urls:
+            title = source.get('title', 'Untitled')
+            url = source.get('url', '')
+            if url:
+                # Use angle brackets to suppress Discord embeds
+                footer += f"- **{title}**: <{url}>\n"
+        return footer
+
+
+async def fetch_and_cache(
+    url: str,
+    web_cache_manager,
+    keywords: Optional[List[str]] = None,
+    keyword_variations: Optional[List[str]] = None,
+    depth: int = 2,
+    fresh: bool = False,
+    progress_callback: Optional[Callable[[str], Awaitable[None]]] = None,
+    user_query: Optional[str] = None
+) -> Optional[Dict]:
+    """
+    Fetch and cache web content using the web cache manager.
+
+    This is a thin wrapper around web_cache_manager.get_or_fetch() that
+    provides a platform-agnostic interface.
+
+    Args:
+        url: URL to fetch
+        web_cache_manager: WebCacheManager instance
+        keywords: Primary keywords for relevance scoring
+        keyword_variations: Keyword variations/synonyms
+        depth: Crawl depth (0 = single page, 1+ = follow links)
+        fresh: If True, bypass cache and fetch fresh
+        progress_callback: Optional async callback for progress updates
+        user_query: Original user query for context
+
+    Returns:
+        Dict with keys: url, title, extracted_text, fetched_at, crawl_depth,
+        source_urls, crawl_tree, token_count, from_cache
+        Returns None if fetch failed
+    """
+    try:
+        result = await web_cache_manager.get_or_fetch(
+            url=url,
+            depth=depth,
+            force_refresh=fresh,
+            query_keywords=keywords if keywords else None,
+            keyword_variations=keyword_variations if keyword_variations else None,
+            progress_callback=progress_callback,
+            user_query=user_query
+        )
+
+        if result:
+            logger.info(f"Web fetch successful: {url} ({len(result.get('source_urls', []))} pages)")
+        else:
+            logger.warning(f"Web fetch failed: {url}")
+
+        return result
+
+    except Exception as e:
+        logger.error(f"Error fetching {url}: {e}", exc_info=True)
+        return None
+
+
+def is_web_fetch_request(content: str) -> bool:
+    """
+    Detect if message content is a web fetch request.
+
+    Requires BOTH a URL AND a fetch keyword (fetch, scrape, crawl).
+    Just having a URL is not enough - user must explicitly request fetch.
+
+    Args:
+        content: Message content to analyze
+
+    Returns:
+        True if this appears to be a web fetch request
+    """
+    url = extract_url_from_content(content)
+    if not url:
+        return False
+    return has_web_fetch_trigger(content)
+
+
+def extract_domain(url: str) -> Optional[str]:
+    """
+    Extract domain from URL using miniuri.
+
+    Args:
+        url: URL to extract domain from
+
+    Returns:
+        Domain string or None if extraction fails
+
+    Example:
+        >>> extract_domain("https://example.com/path")
+        'example.com'
+    """
+    try:
+        uri = Uri(url)
+        return uri.authority or None
+    except Exception as e:
+        logger.warning(f"Failed to extract domain from {url}: {e}")
+        # Fallback to urlparse
+        from urllib.parse import urlparse
+        try:
+            return urlparse(url).netloc or None
+        except:
+            return None
+
+
+def format_progress_message(
+    url: str,
+    task: str = "",
+    keywords: Optional[List[str]] = None,
+    variations: Optional[List[str]] = None,
+    stems: Optional[List[str]] = None,
+    format_instruction: str = "",
+    depth: int = 2,
+    cache_mode: str = "default",
+    status: str = "",
+    crawl_delay: Optional[float] = None
+) -> str:
+    """
+    Format a detailed progress message for web fetch operations.
+
+    Args:
+        url: URL being fetched
+        task: Extracted task description
+        keywords: Primary keywords
+        variations: Keyword variations
+        stems: Keyword stems
+        format_instruction: Format instructions
+        depth: Crawl depth
+        cache_mode: "default" or "fresh"
+        status: Current status message
+        crawl_delay: Optional crawl delay in seconds
+
+    Returns:
+        Formatted progress message
+    """
+    task_display = task if task else 'none'
+    keywords_display = ', '.join(keywords) if keywords else 'none'
+    variations_display = ', '.join(variations) if variations else 'none'
+    stems_display = ', '.join(stems) if stems else 'none'
+    format_display = format_instruction if format_instruction else 'default'
+
+    msg = f"🔍 **Searching** <{url}>\n"
+    msg += f"**Task:** {task_display}\n"
+    msg += f"**Keywords:** {keywords_display}\n"
+    msg += f"**Variations:** {variations_display}\n"
+    msg += f"**Stems:** {stems_display}\n"
+    msg += f"**Format:** {format_display}\n"
+    msg += f"**Depth:** {depth}\n"
+    msg += f"**Mode:** {cache_mode}"
+
+    if crawl_delay and crawl_delay > 2.0:
+        msg += f"\n**Crawl delay:** {crawl_delay}s (from robots.txt)"
+
+    if status:
+        msg += f"\n\n{status}"
+
+    return msg
+
+
+def format_answering_message(num_pages: int) -> str:
+    """
+    Format a message for the answering phase after fetch completes.
+
+    Args:
+        num_pages: Number of pages fetched
+
+    Returns:
+        Formatted answering message
+    """
+    return f"💭 **Answering your research...**\nAnalyzing {num_pages} page(s)"
+
+
+def generate_keyword_stems(keywords: List[str], variations: List[str]) -> List[str]:
+    """
+    Generate stems from keywords and variations using simple suffix stripping.
+
+    This is a copy of the function from bot.py to avoid circular imports.
+
+    Args:
+        keywords: Primary keywords
+        variations: Keyword variations
+
+    Returns:
+        List of unique stems (excluding duplicates from keywords/variations)
+    """
+    def simple_stem(word: str) -> str:
+        """Simple suffix-stripping stemmer for common English suffixes"""
+        word = word.lower()
+        # Common suffixes in order of longest first
+        suffixes = ['ings', 'ing', 'ed', 'es', 's', 'ly', 'er', 'est', 'tion', 'sion', 'ness', 'ment', 'able', 'ible']
+        for suffix in suffixes:
+            if word.endswith(suffix) and len(word) > len(suffix) + 2:  # Keep at least 3 chars in stem
+                return word[:-len(suffix)]
+        return word
+
+    stems = []
+    # Lowercase all keywords and variations for comparison
+    keywords_lower = [kw.lower() for kw in keywords]
+    variations_lower = [v.lower() for v in variations]
+
+    all_words = keywords + variations
+    for word in all_words:
+        # Split phrases into individual words and stem each separately
+        individual_words = word.split()
+        for individual_word in individual_words:
+            stem = simple_stem(individual_word)
+            # Only add if:
+            # 1. Stem is different from original word (stemming actually changed it)
+            # 2. Stem is not already in keywords (case-insensitive)
+            # 3. Stem is not already in variations (case-insensitive)
+            if (stem != individual_word.lower() and
+                stem not in keywords_lower and
+                stem not in variations_lower):
+                stems.append(stem)
+
+    # Deduplicate stems
+    return list(dict.fromkeys(stems))

pyproject.toml

@@ -32,11 +32,29 @@ mesh = [
   # cryptography (also core). This extras block exists as the documented
   # opt-in surface even though no extra packages are required today.
 ]
+crawler = [
+  # Verbatim lift from agents.ai.unturf.com/core. Off by default — the
+  # default test suite never imports the crawler. Install with:
+  #     pip install 'aborist[crawler]'
+  # then run `make test-crawler`.
+  "aiohttp>=3.8",
+  "beautifulsoup4>=4.11",
+  "lxml>=4.9",
+  "html5lib>=1.1",
+  "html2text>=2024.2.26",
+  "miniuri>=1.1",
+  "feedparser>=6.0",
+  "Pillow>=10.0",
+  "cairosvg>=2.7",
+  "pypdf>=4.0",
+]
 dev = [
   "pytest>=8",
+  "pytest-asyncio>=0.23",
   "aborist[html]",
   "aborist[wikitext]",
   "aborist[mesh]",
+  "aborist[crawler]",
 ]
 
 [project.scripts]

tests/crawler/test_async_web_fetcher.py

@@ -0,0 +1,470 @@
+"""
+Tests for async_web_fetcher module.
+
+Tests URL handling, media extraction, robots.txt compliance,
+and async fetching functionality.
+"""
+
+import pytest
+from unittest.mock import Mock, AsyncMock, patch, MagicMock
+import asyncio
+import aiohttp
+
+from aborist.sources.crawler.async_web_fetcher import (
+    CrawlMode,
+    MediaItem,
+    strip_uri_fragment,
+    normalize_link,
+    get_media_type_from_extension,
+    get_media_type_from_mime,
+    extract_media_from_html,
+    AsyncWebFetcher,
+    IMAGE_EXTENSIONS,
+    VIDEO_EXTENSIONS,
+    AUDIO_EXTENSIONS,
+    DEFAULT_CRAWL_DELAY,
+    get_last_fetch_error,
+    clear_last_fetch_error,
+)
+
+
+class TestCrawlMode:
+    """Test CrawlMode enum."""
+
+    def test_crawl_mode_values(self):
+        """Test all crawl mode values exist."""
+        assert CrawlMode.TEXT.value == "text"
+        assert CrawlMode.IMAGES.value == "images"
+        assert CrawlMode.VIDEOS.value == "videos"
+        assert CrawlMode.MEDIA.value == "media"
+        assert CrawlMode.ALL.value == "all"
+
+    def test_crawl_mode_from_string(self):
+        """Test creating CrawlMode from string."""
+        assert CrawlMode("text") == CrawlMode.TEXT
+        assert CrawlMode("images") == CrawlMode.IMAGES
+        assert CrawlMode("media") == CrawlMode.MEDIA
+
+
+class TestMediaItem:
+    """Test MediaItem dataclass."""
+
+    def test_media_item_creation(self):
+        """Test creating a MediaItem."""
+        item = MediaItem(
+            url="https://example.com/image.jpg",
+            source_page="https://example.com/",
+            media_type="image",
+            mime_type="image/jpeg",
+        )
+        assert item.url == "https://example.com/image.jpg"
+        assert item.source_page == "https://example.com/"
+        assert item.media_type == "image"
+        assert item.mime_type == "image/jpeg"
+        assert item.discovered_at is not None
+
+    def test_media_item_optional_fields(self):
+        """Test MediaItem with optional fields."""
+        item = MediaItem(
+            url="https://example.com/video.mp4",
+            source_page="https://example.com/",
+            media_type="video",
+            alt_text="A video",
+            width=1920,
+            height=1080,
+        )
+        assert item.alt_text == "A video"
+        assert item.width == 1920
+        assert item.height == 1080
+
+
+class TestURIHandling:
+    """Test URI handling functions."""
+
+    def test_strip_uri_fragment_basic(self):
+        """Test stripping fragment from URI."""
+        uri = "https://example.com/page#section"
+        result = strip_uri_fragment(uri)
+        assert result == "https://example.com/page"
+
+    def test_strip_uri_fragment_no_fragment(self):
+        """Test URI without fragment."""
+        uri = "https://example.com/page"
+        result = strip_uri_fragment(uri)
+        assert result == "https://example.com/page"
+
+    def test_strip_uri_fragment_with_query(self):
+        """Test URI with query and fragment."""
+        uri = "https://example.com/page?q=test#section"
+        result = strip_uri_fragment(uri)
+        assert result == "https://example.com/page?q=test"
+
+    def test_strip_uri_fragment_empty(self):
+        """Test empty URI."""
+        assert strip_uri_fragment("") == ""
+        assert strip_uri_fragment(None) is None
+
+    def test_normalize_link_dict(self):
+        """Test normalizing link from dict."""
+        link = {"url": "https://example.com/page#section", "anchor_text": "Click here"}
+        uri, anchor = normalize_link(link)
+        assert uri == "https://example.com/page"
+        assert anchor == "Click here"
+
+    def test_normalize_link_string(self):
+        """Test normalizing link from string."""
+        uri, anchor = normalize_link("https://example.com/page#section")
+        assert uri == "https://example.com/page"
+        assert anchor == ""
+
+
+class TestMediaTypeDetection:
+    """Test media type detection functions."""
+
+    def test_get_media_type_from_extension_image(self):
+        """Test detecting image from extension."""
+        assert get_media_type_from_extension("https://example.com/photo.jpg") == "image"
+        assert get_media_type_from_extension("https://example.com/photo.png") == "image"
+        assert get_media_type_from_extension("https://example.com/photo.webp") == "image"
+        assert get_media_type_from_extension("https://example.com/photo.gif") == "image"
+
+    def test_get_media_type_from_extension_video(self):
+        """Test detecting video from extension."""
+        assert get_media_type_from_extension("https://example.com/video.mp4") == "video"
+        assert get_media_type_from_extension("https://example.com/video.webm") == "video"
+        assert get_media_type_from_extension("https://example.com/video.mov") == "video"
+
+    def test_get_media_type_from_extension_audio(self):
+        """Test detecting audio from extension."""
+        assert get_media_type_from_extension("https://example.com/audio.mp3") == "audio"
+        assert get_media_type_from_extension("https://example.com/audio.wav") == "audio"
+        assert get_media_type_from_extension("https://example.com/audio.ogg") == "audio"
+
+    def test_get_media_type_from_extension_code(self):
+        """Test detecting code from extension."""
+        assert get_media_type_from_extension("https://example.com/file.txt") == "code"
+        assert get_media_type_from_extension("https://example.com/script.py") == "code"
+        assert get_media_type_from_extension("https://example.com/app.js") == "code"
+
+    def test_get_media_type_from_extension_unknown(self):
+        """Test unknown extension."""
+        assert get_media_type_from_extension("https://example.com/page.html") is None
+        assert get_media_type_from_extension("https://example.com/file.xyz") is None
+
+    def test_get_media_type_from_mime_image(self):
+        """Test detecting image from MIME type."""
+        assert get_media_type_from_mime("image/jpeg") == "image"
+        assert get_media_type_from_mime("image/png") == "image"
+        assert get_media_type_from_mime("image/webp") == "image"
+
+    def test_get_media_type_from_mime_video(self):
+        """Test detecting video from MIME type."""
+        assert get_media_type_from_mime("video/mp4") == "video"
+        assert get_media_type_from_mime("video/webm") == "video"
+
+    def test_get_media_type_from_mime_audio(self):
+        """Test detecting audio from MIME type."""
+        assert get_media_type_from_mime("audio/mpeg") == "audio"
+        assert get_media_type_from_mime("audio/wav") == "audio"
+
+    def test_get_media_type_from_mime_code(self):
+        """Test detecting code from MIME type."""
+        assert get_media_type_from_mime("application/json") == "code"
+        assert get_media_type_from_mime("text/plain") == "code"
+        assert get_media_type_from_mime("application/javascript") == "code"
+
+    def test_get_media_type_from_mime_unknown(self):
+        """Test unknown MIME type."""
+        assert get_media_type_from_mime("text/html") is None
+        assert get_media_type_from_mime("application/octet-stream") is None
+
+
+class TestMediaExtraction:
+    """Test media extraction from HTML."""
+
+    def test_extract_images_from_html(self):
+        """Test extracting images from HTML."""
+        html = '''
+        <html>
+        <body>
+            <img src="/images/photo.jpg" alt="A photo">
+            <img src="https://cdn.example.com/logo.png" alt="Logo">
+        </body>
+        </html>
+        '''
+        media = extract_media_from_html(html, "https://example.com/", CrawlMode.IMAGES)
+
+        assert len(media) >= 1
+        urls = [m['url'] for m in media]
+        # Should have resolved relative URL
+        assert any('photo.jpg' in url for url in urls)
+
+    def test_extract_videos_from_html(self):
+        """Test extracting videos from HTML."""
+        html = '''
+        <html>
+        <body>
+            <video src="/videos/movie.mp4"></video>
+            <source src="/videos/clip.webm" type="video/webm">
+        </body>
+        </html>
+        '''
+        media = extract_media_from_html(html, "https://example.com/", CrawlMode.VIDEOS)
+
+        urls = [m['url'] for m in media]
+        assert any('movie.mp4' in url or 'clip.webm' in url for url in urls)
+
+    def test_extract_media_filters_by_mode(self):
+        """Test that extraction respects crawl mode."""
+        html = '''
+        <html>
+        <body>
+            <img src="/photo.jpg">
+            <video src="/video.mp4"></video>
+            <audio src="/audio.mp3"></audio>
+        </body>
+        </html>
+        '''
+
+        # Images only
+        images = extract_media_from_html(html, "https://example.com/", CrawlMode.IMAGES)
+        image_urls = [m['url'] for m in images]
+        assert any('photo.jpg' in url for url in image_urls)
+        assert not any('video.mp4' in url for url in image_urls)
+
+        # Videos only
+        videos = extract_media_from_html(html, "https://example.com/", CrawlMode.VIDEOS)
+        video_urls = [m['url'] for m in videos]
+        assert any('video.mp4' in url for url in video_urls)
+
+
+class TestAsyncWebFetcher:
+    """Test AsyncWebFetcher class."""
+
+    def test_fetcher_initialization(self):
+        """Test AsyncWebFetcher initialization."""
+        fetcher = AsyncWebFetcher(
+            user_agent="TestBot/1.0",
+            default_crawl_delay=1.0,
+        )
+        assert fetcher.user_agent == "TestBot/1.0"
+        assert fetcher.default_crawl_delay == 1.0
+
+    def test_fetcher_default_values(self):
+        """Test AsyncWebFetcher default values."""
+        fetcher = AsyncWebFetcher()
+        assert fetcher.default_crawl_delay == DEFAULT_CRAWL_DELAY
+
+    def test_get_domain(self):
+        """Test domain extraction."""
+        fetcher = AsyncWebFetcher()
+        assert fetcher._get_domain("https://example.com/page") == "example.com"
+        assert fetcher._get_domain("https://sub.example.com/page") == "sub.example.com"
+
+    def test_simple_stem(self):
+        """Test simple word stemming."""
+        fetcher = AsyncWebFetcher()
+
+        # Test stemming - actual behavior may vary
+        stem = fetcher._simple_stem("running")
+        assert "run" in stem  # Should contain "run"
+
+        # Short words should not be stemmed
+        assert fetcher._simple_stem("go") == "go"
+
+    def test_extract_text_content(self):
+        """Test text extraction from HTML."""
+        fetcher = AsyncWebFetcher()
+        html = '''
+        <html>
+        <head><title>Test Page</title></head>
+        <body>
+            <h1>Hello World</h1>
+            <p>This is a test paragraph.</p>
+            <script>alert('ignored');</script>
+            <style>.hidden { display: none; }</style>
+        </body>
+        </html>
+        '''
+        text = fetcher.extract_text_content(html)
+
+        assert "Hello World" in text
+        assert "test paragraph" in text
+        assert "alert" not in text  # Script content should be removed
+        assert ".hidden" not in text  # Style content should be removed
+
+    def test_extract_title(self):
+        """Test title extraction from HTML."""
+        fetcher = AsyncWebFetcher()
+        html = '<html><head><title>Page Title</title></head><body></body></html>'
+        title = fetcher.extract_title(html)
+        assert title == "Page Title"
+
+    def test_extract_title_missing(self):
+        """Test title extraction when missing."""
+        fetcher = AsyncWebFetcher()
+        html = '<html><head></head><body></body></html>'
+        title = fetcher.extract_title(html)
+        # Returns "Untitled" when no title present
+        assert title == "Untitled" or title == ""
+
+
+class TestAsyncWebFetcherScoring:
+    """Test link and page scoring functionality."""
+
+    def test_score_link_keyword_match(self):
+        """Test link scoring with keyword matches."""
+        fetcher = AsyncWebFetcher()
+
+        # Link with keyword in URL
+        score = fetcher._score_link(
+            "https://example.com/pricing",
+            "View Pricing",
+            query_keywords=["pricing", "cost"],
+        )
+        assert score > 0  # Should have positive score
+
+        # Link without keywords
+        score_none = fetcher._score_link(
+            "https://example.com/about",
+            "About Us",
+            query_keywords=["pricing", "cost"],
+        )
+        assert score > score_none  # Keyword match should score higher
+
+    def test_score_link_anchor_text(self):
+        """Test link scoring with anchor text."""
+        fetcher = AsyncWebFetcher()
+
+        # Anchor text with keyword
+        score = fetcher._score_link(
+            "https://example.com/page",
+            "Our Pricing Plans",
+            query_keywords=["pricing"],
+        )
+        assert score > 0
+
+    def test_score_page_keyword_match(self):
+        """Test page scoring with keyword matches."""
+        fetcher = AsyncWebFetcher()
+
+        page_data = {
+            "url": "https://example.com/pricing",
+            "title": "Pricing Plans",
+            "extracted_text": "Our pricing is competitive. See our cost breakdown.",
+        }
+
+        score = fetcher._score_page(
+            page_data,
+            query_keywords=["pricing", "cost"],
+        )
+        assert score > 0
+
+
+class TestAsyncWebFetcherAsync:
+    """Test async methods of AsyncWebFetcher."""
+
+    @pytest.mark.asyncio
+    async def test_can_fetch_respects_robots(self):
+        """Test that _can_fetch respects robots.txt."""
+        fetcher = AsyncWebFetcher()
+
+        # Mock the robots.txt fetch to return a parser that disallows
+        with patch.object(fetcher, '_fetch_robots_txt', new_callable=AsyncMock) as mock_robots:
+            mock_parser = MagicMock()
+            mock_parser.can_fetch.return_value = False
+            mock_robots.return_value = mock_parser
+
+            result = await fetcher._can_fetch("https://example.com/blocked")
+            assert result is False
+
+    @pytest.mark.asyncio
+    async def test_can_fetch_allows_when_parser_allows(self):
+        """Test _can_fetch when robots parser allows."""
+        fetcher = AsyncWebFetcher()
+
+        with patch.object(fetcher, '_fetch_robots_txt', new_callable=AsyncMock) as mock_robots:
+            mock_parser = MagicMock()
+            mock_parser.can_fetch.return_value = True
+            mock_robots.return_value = mock_parser
+
+            result = await fetcher._can_fetch("https://example.com/allowed")
+            assert result is True
+
+    @pytest.mark.asyncio
+    async def test_enforce_crawl_delay(self):
+        """Test crawl delay enforcement."""
+        fetcher = AsyncWebFetcher(default_crawl_delay=0.1)
+        domain = "example.com"
+
+        # First request should not delay
+        start = asyncio.get_event_loop().time()
+        await fetcher._enforce_crawl_delay(domain)
+        first_time = asyncio.get_event_loop().time() - start
+
+        # Second request should delay
+        start = asyncio.get_event_loop().time()
+        await fetcher._enforce_crawl_delay(domain)
+        second_time = asyncio.get_event_loop().time() - start
+
+        # Second call should have some delay (at least part of 0.1s)
+        assert second_time >= 0.05  # Allow some tolerance
+
+    @pytest.mark.asyncio
+    async def test_check_media_url_signature(self):
+        """Test check_media_url function signature."""
+        fetcher = AsyncWebFetcher()
+
+        # Verify the method exists and has correct signature
+        import inspect
+        sig = inspect.signature(fetcher.check_media_url)
+        params = list(sig.parameters.keys())
+
+        # Should have url parameter
+        assert 'url' in params or len(params) >= 1
+
+
+class TestErrorHandling:
+    """Test error handling functionality."""
+
+    def test_get_last_fetch_error_initially_none(self):
+        """Test that last fetch error is None initially."""
+        clear_last_fetch_error()
+        error = get_last_fetch_error()
+        assert error is None
+
+    def test_clear_last_fetch_error(self):
+        """Test clearing the last fetch error."""
+        clear_last_fetch_error()
+        assert get_last_fetch_error() is None
+
+
+class TestExtensionSets:
+    """Test extension and MIME type sets."""
+
+    def test_image_extensions(self):
+        """Test image extension set."""
+        assert '.jpg' in IMAGE_EXTENSIONS
+        assert '.jpeg' in IMAGE_EXTENSIONS
+        assert '.png' in IMAGE_EXTENSIONS
+        assert '.gif' in IMAGE_EXTENSIONS
+        assert '.webp' in IMAGE_EXTENSIONS
+        assert '.svg' in IMAGE_EXTENSIONS
+
+    def test_video_extensions(self):
+        """Test video extension set."""
+        assert '.mp4' in VIDEO_EXTENSIONS
+        assert '.webm' in VIDEO_EXTENSIONS
+        assert '.mov' in VIDEO_EXTENSIONS
+        assert '.avi' in VIDEO_EXTENSIONS
+
+    def test_audio_extensions(self):
+        """Test audio extension set."""
+        assert '.mp3' in AUDIO_EXTENSIONS
+        assert '.wav' in AUDIO_EXTENSIONS
+        assert '.ogg' in AUDIO_EXTENSIONS
+        assert '.flac' in AUDIO_EXTENSIONS
+
+
+if __name__ == '__main__':
+    pytest.main([__file__, '-v'])

tests/crawler/test_web_fetch.py

@@ -0,0 +1,343 @@
+"""
+Tests for core.web_fetch module.
+
+Tests URL extraction, sources footer building, and web fetch helpers.
+"""
+
+import pytest
+from unittest.mock import Mock, AsyncMock, patch
+from aborist.sources.crawler.web_fetch import (
+    extract_url_from_content,
+    is_web_fetch_request,
+    build_sources_footer,
+    detect_fresh_fetch_intent,
+    extract_domain,
+    format_progress_message,
+    format_answering_message,
+    generate_keyword_stems,
+)
+
+
+class TestURLExtraction:
+    """Test URL extraction from message content."""
+
+    def test_extract_url_with_protocol(self):
+        """Test extracting URL with http:// or https://"""
+        content = "Check out https://example.com for more info"
+        url = extract_url_from_content(content)
+        assert url == "https://example.com"
+
+    def test_extract_url_http_protocol(self):
+        """Test extracting URL with http:// protocol"""
+        content = "Visit http://insecure.example.com"
+        url = extract_url_from_content(content)
+        assert url == "http://insecure.example.com"
+
+    def test_extract_url_without_protocol(self):
+        """Test extracting URL without protocol (adds https://)"""
+        content = "Go to example.com"
+        url = extract_url_from_content(content)
+        assert url == "https://example.com"
+
+    def test_extract_subdomain_url(self):
+        """Test extracting URL with subdomain"""
+        content = "Check media.unturf.com/test"
+        url = extract_url_from_content(content)
+        assert url == "https://media.unturf.com/test"
+
+    def test_extract_url_with_path(self):
+        """Test extracting URL with path"""
+        content = "Read https://docs.example.com/api/v1/reference"
+        url = extract_url_from_content(content)
+        assert url == "https://docs.example.com/api/v1/reference"
+
+    def test_extract_url_with_query(self):
+        """Test extracting URL with query parameters"""
+        content = "See https://example.com/search?q=test&lang=en"
+        url = extract_url_from_content(content)
+        assert url == "https://example.com/search?q=test&lang=en"
+
+    def test_no_url_in_content(self):
+        """Test when no URL is present"""
+        content = "This is just a regular message without any URLs"
+        url = extract_url_from_content(content)
+        assert url is None
+
+    def test_extract_first_url_when_multiple(self):
+        """Test that first URL is extracted when multiple are present"""
+        content = "Check https://first.com and https://second.com"
+        url = extract_url_from_content(content)
+        assert url == "https://first.com"
+
+    def test_extract_url_with_www(self):
+        """Test extracting URL with www prefix"""
+        content = "Visit www.example.com"
+        url = extract_url_from_content(content)
+        assert url == "https://www.example.com"
+
+    def test_extract_url_ignores_discord_embeds(self):
+        """Test that angle brackets don't break URL extraction"""
+        content = "Check out <https://example.com>"
+        url = extract_url_from_content(content)
+        # Angle brackets should be part of the URL match
+        assert "example.com" in url
+
+
+@pytest.mark.skip(
+    reason="chat-bot fetch triggers (has_web_fetch_trigger) were "
+    "dropped during the aborist lift — aborist has no chat surface; "
+    "see aborist/sources/crawler/__init__.py for rationale."
+)
+class TestWebFetchDetection:
+    """Test web fetch request detection."""
+
+    def test_is_web_fetch_request_with_url(self):
+        """Test detection when URL is present"""
+        assert is_web_fetch_request("Fetch https://example.com") is True
+
+    def test_is_web_fetch_request_without_url(self):
+        """Test detection when no URL is present"""
+        assert is_web_fetch_request("Tell me about Python") is False
+
+    def test_is_web_fetch_request_domain_only(self):
+        """Test detection with domain-only URL"""
+        assert is_web_fetch_request("What's on example.com?") is True
+
+
+@pytest.mark.skip(
+    reason="chat-bot fresh-fetch triggers (has_fresh_fetch_trigger) were "
+    "dropped during the aborist lift — aborist has no chat surface; "
+    "see aborist/sources/crawler/__init__.py for rationale."
+)
+class TestFreshFetchDetection:
+    """Test fresh fetch intent detection."""
+
+    def test_detect_fresh_with_keyword(self):
+        """Test detection with 'fresh' keyword"""
+        assert detect_fresh_fetch_intent("Get fresh data from example.com") is True
+
+    def test_detect_refresh_with_keyword(self):
+        """Test detection with 'refresh' keyword"""
+        assert detect_fresh_fetch_intent("Refresh the content") is True
+
+    def test_detect_reload_with_keyword(self):
+        """Test detection with 'reload' keyword"""
+        assert detect_fresh_fetch_intent("Reload example.com") is True
+
+    def test_detect_latest_with_keyword(self):
+        """Test detection with 'latest' keyword"""
+        assert detect_fresh_fetch_intent("Get the latest from example.com") is True
+
+    def test_no_fresh_intent(self):
+        """Test when no fresh fetch keywords present"""
+        assert detect_fresh_fetch_intent("Fetch example.com") is False
+
+    def test_fresh_detection_case_insensitive(self):
+        """Test that fresh detection is case insensitive"""
+        assert detect_fresh_fetch_intent("FRESH data please") is True
+
+
+class TestSourcesFooter:
+    """Test sources footer building."""
+
+    def test_build_sources_footer_single_source(self):
+        """Test building footer with single source"""
+        sources = [{'url': 'https://example.com', 'title': 'Example Page'}]
+        footer = build_sources_footer(sources)
+        assert 'Sources:' in footer
+        assert 'Example Page' in footer
+        assert '<https://example.com>' in footer
+
+    def test_build_sources_footer_multiple_sources(self):
+        """Test building footer with multiple sources"""
+        sources = [
+            {'url': 'https://example.com', 'title': 'Example Page'},
+            {'url': 'https://test.com', 'title': 'Test Page'}
+        ]
+        footer = build_sources_footer(sources)
+        assert 'Example Page' in footer
+        assert 'Test Page' in footer
+        assert footer.count('<') == 2  # Two URLs with angle brackets
+
+    def test_build_sources_footer_empty_list(self):
+        """Test building footer with empty source list"""
+        footer = build_sources_footer([])
+        assert footer == ""
+
+    def test_build_sources_footer_missing_title(self):
+        """Test building footer when title is missing"""
+        sources = [{'url': 'https://example.com'}]
+        footer = build_sources_footer(sources)
+        assert 'Untitled' in footer
+        assert '<https://example.com>' in footer
+
+    def test_build_sources_footer_missing_url(self):
+        """Test building footer when URL is missing"""
+        sources = [{'title': 'Example Page'}]
+        footer = build_sources_footer(sources)
+        # Should not include entry if URL is missing
+        assert footer == '**Sources:**\n'
+
+    def test_build_sources_footer_angle_brackets_suppress_embeds(self):
+        """Test that angle brackets are used to suppress Discord embeds"""
+        sources = [{'url': 'https://example.com', 'title': 'Test'}]
+        footer = build_sources_footer(sources)
+        # URLs should be in angle brackets
+        assert '<https://example.com>' in footer
+        assert 'https://example.com>' not in footer.replace('<https://example.com>', '')
+
+
+class TestDomainExtraction:
+    """Test domain extraction from URLs."""
+
+    def test_extract_domain_basic(self):
+        """Test extracting domain from basic URL"""
+        domain = extract_domain("https://example.com/path")
+        assert domain == "example.com"
+
+    def test_extract_domain_with_subdomain(self):
+        """Test extracting domain with subdomain"""
+        domain = extract_domain("https://api.example.com")
+        assert domain == "api.example.com"
+
+    def test_extract_domain_with_port(self):
+        """Test extracting domain with port"""
+        domain = extract_domain("https://example.com:8080/path")
+        assert domain == "example.com:8080"
+
+    def test_extract_domain_invalid_url(self):
+        """Test extracting domain from invalid URL"""
+        domain = extract_domain("not a url")
+        # Should return None or handle gracefully
+        assert domain is None or isinstance(domain, str)
+
+
+class TestProgressFormatting:
+    """Test progress message formatting."""
+
+    def test_format_progress_message_basic(self):
+        """Test basic progress message formatting"""
+        msg = format_progress_message(
+            url="https://example.com",
+            task="Find pricing",
+            keywords=["pricing", "cost"],
+            depth=2
+        )
+        assert "example.com" in msg
+        assert "pricing" in msg
+        assert "cost" in msg
+        assert "**Depth:** 2" in msg
+
+    def test_format_progress_message_with_crawl_delay(self):
+        """Test progress message with crawl delay"""
+        msg = format_progress_message(
+            url="https://example.com",
+            crawl_delay=5.0
+        )
+        assert "**Crawl delay:** 5.0s" in msg
+
+    def test_format_progress_message_no_crawl_delay_if_default(self):
+        """Test that default crawl delay is not shown"""
+        msg = format_progress_message(
+            url="https://example.com",
+            crawl_delay=2.0
+        )
+        assert "Crawl delay" not in msg
+
+    def test_format_progress_message_fresh_mode(self):
+        """Test progress message with fresh mode"""
+        msg = format_progress_message(
+            url="https://example.com",
+            cache_mode="fresh"
+        )
+        assert "**Mode:** fresh" in msg
+
+    def test_format_answering_message(self):
+        """Test answering phase message formatting"""
+        msg = format_answering_message(5)
+        assert "5 page(s)" in msg
+        assert "Analyzing" in msg
+
+
+class TestKeywordStemming:
+    """Test keyword stemming functionality."""
+
+    def test_generate_stems_basic(self):
+        """Test generating stems from keywords"""
+        stems = generate_keyword_stems(["running", "tests"], [])
+        assert "run" in stems or "test" in stems
+
+    def test_generate_stems_no_duplicates(self):
+        """Test that stems don't duplicate keywords"""
+        stems = generate_keyword_stems(["run"], ["running"])
+        # "run" is already in keywords, so stem shouldn't be added
+        assert "run" not in stems
+
+    def test_generate_stems_variations(self):
+        """Test stem generation from variations"""
+        stems = generate_keyword_stems([], ["pricing", "prices"])
+        # Should generate stems if they're different from variations
+        assert isinstance(stems, list)
+
+    def test_generate_stems_no_stemming_needed(self):
+        """Test when no stemming is needed"""
+        stems = generate_keyword_stems(["api", "cli"], [])
+        # Short words shouldn't be stemmed
+        assert len(stems) == 0
+
+    def test_generate_stems_phrase(self):
+        """Test stem generation from phrase"""
+        stems = generate_keyword_stems(["pull requests"], [])
+        # Should split phrase and stem individual words
+        assert isinstance(stems, list)
+
+
+class TestWebFetchIntegration:
+    """Test web fetch integration with mock web cache manager."""
+
+    @pytest.mark.asyncio
+    async def test_fetch_and_cache_success(self):
+        """Test successful fetch and cache operation"""
+        # This would require mocking web_cache_manager
+        # For now, just verify the function exists and has correct signature
+        from aborist.sources.crawler.web_fetch import fetch_and_cache
+        import inspect
+        sig = inspect.signature(fetch_and_cache)
+        assert 'url' in sig.parameters
+        assert 'web_cache_manager' in sig.parameters
+        assert 'keywords' in sig.parameters
+        assert 'depth' in sig.parameters
+        assert 'fresh' in sig.parameters
+
+    @pytest.mark.asyncio
+    async def test_fetch_and_cache_with_progress_callback(self):
+        """Test fetch with progress callback"""
+        from aborist.sources.crawler.web_fetch import fetch_and_cache
+
+        # Mock web cache manager
+        mock_manager = Mock()
+        mock_manager.get_or_fetch = AsyncMock(return_value={
+            'url': 'https://example.com',
+            'title': 'Test',
+            'extracted_text': 'Content',
+            'source_urls': []
+        })
+
+        # Mock progress callback
+        progress_calls = []
+        async def progress_callback(msg):
+            progress_calls.append(msg)
+
+        result = await fetch_and_cache(
+            url='https://example.com',
+            web_cache_manager=mock_manager,
+            progress_callback=progress_callback
+        )
+
+        assert result is not None
+        assert result['url'] == 'https://example.com'
+        mock_manager.get_or_fetch.assert_called_once()
+
+
+if __name__ == '__main__':
+    pytest.main([__file__, '-v'])