010: Ethical crawl mode for site indexing ($14/m plan)

Reporter: fxhp Date: 2026-03-03 Priority: medium Status: open Affects: Browser extension (paid tier)

Description

Paid feature ($14/m) that crawls a site following robots.txt rules, indexing pages for keyword search and cross-page question answering. All data stored in encrypted localStorage on the user's device.

Goal

Users can index an entire site (or section) without manually opening the chatbot on every page. Then ask questions across all indexed pages like "which page talks about authentication?" or "find all mentions of pricing."

Architecture

User triggers crawl from the chatbot UI (new "Index Site" button in settings)
Extension fetches robots.txt, parses disallow/allow rules
Starts from current page, follows internal links breadth-first
For each page: extract content, compute page intelligence, store summary
All data encrypted in vault localStorage
Respect: rate limiting (1 req/sec), max depth, max pages, robots.txt
Progress UI shows pages indexed, estimated coverage

Ethics

Strictly follows robots.txt (no crawl if disallowed)
Rate limited to 1 request per second minimum
User-initiated only (never auto-crawl)
All data stays on the user's device
Clear disclosure: "This indexes pages for your private use only"

Monetization

Free tier: manual page-by-page indexing (open chatbot on each page)
$14/m plan: automated crawl mode + higher page limits + priority support
Payment via unsandbox.com accounts

Dependencies

Reverse RAG page knowledge (#006)
Vault encryption (implemented)
unsandbox.com billing integration

1.6 KiB Raw Permalink Blame History