implement reverse RAG, journey-aware greetings, vault viewer, crawl mode, unfirehose sharing (#004-#012)

This commit is contained in:
russell@unturf.com 2026-03-03 17:46:55 -05:00
parent fa0ad2782d
commit a0a2327af6
14 changed files with 1363 additions and 29 deletions

View file

@ -3,7 +3,7 @@
**Reporter:** cthegray
**Date:** 2026-03-03
**Priority:** high
**Status:** open
**Status:** fixed
**Affects:** Browser extension (Chrome/Safari)
## Description
@ -36,10 +36,17 @@ The browser-toys extension injects `<script src="https://uncloseai.com/uncloseai
## Fix
In the `uncloseai-browser-toys` repo, switch the content script from DOM script injection to `chrome.scripting.executeScript()` with the bundled IIFE (`uncloseai-bundle.js`). Extensions executing their own bundled code bypass page CSP entirely.
Implemented three-tier injection strategy in `uncloseai-browser-toys`:
The bundle is ready: `make bundle-extension` produces `public/uncloseai-bundle.js` (1.6MB IIFE, CSP-safe, no ES modules, no CDN imports).
1. **Strategy 1 (CSP-safe):** Content script messages the background service worker, which calls `chrome.scripting.executeScript({ world: "MAIN", files: ["uncloseai-bundle.js"] })`. This injects directly into the page world via the browser engine, bypassing all page CSP restrictions. Requires `"scripting"` permission (added to Chrome and Safari MV3 manifests).
## Notes
2. **Strategy 2 (script tag):** Fallback for MV2 browsers (Firefox). Creates a `<script>` tag with `src` pointing to the extension-origin bundle. Works on most pages since browsers whitelist extension origins in CSP.
Fix lives in the browser-toys repo, not this one. The bundle and architecture are ready here.
3. **Strategy 3 (CDN bootstrap):** Last resort if extension resources fail. Inlines a bootstrap that loads from `uncloseai.com/uncloseai.js`.
**Files changed in browser-toys:**
- `shared/content.js`: Rewrote injection to try background message first, then script tag, then CDN
- `shared/background.js`: Added `onMessage` listener for `{ action: "inject" }` using `chrome.scripting.executeScript`
- `extensions/chrome/manifest.json`: Added `"scripting"` permission
- `extensions/safari/manifest.json`: Added `"scripting"` permission
- `extensions/firefox/manifest.json`: No changes needed (MV2, uses Strategy 2)

View file

@ -0,0 +1,37 @@
# 006: Reverse RAG: cross-page context via encrypted localStorage
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** medium
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
When a user browses multiple pages on the same site, the machine learning model should be able to reference content from previously visited pages. Currently each page is isolated: the model only knows the current page's content.
## Goal
Build a reverse RAG system where page summaries are stored in encrypted localStorage (per-domain), and automatically included in the system prompt for new pages. This gives the model a growing knowledge base about the site as the user browses.
## Architecture
1. After generating the intro message for a page, extract a brief summary (title, key topics, 2-3 sentences)
2. Store summaries per-domain in vault-encrypted localStorage: `uncloseai-pageknowledge-{domain}`
3. On new pages, include the last N page summaries in the system prompt as context
4. Summaries should be compact (under 200 tokens each) to avoid context overflow
5. Use the existing `fitPageContent()` budget system to allocate space for cross-page context
## Privacy
All cross-page data must be encrypted via the vault, same as chat history. The user controls their data. No data leaves the device except through explicit LLM requests.
## Dependencies
- Site journey tracking (implemented in storage.js)
- Vault encryption (already working)
- `fitPageContent()` context budgeting (already working)
## Notes
This is the foundation for making uncloseai "unstoppable" on any site. The model gets smarter the more pages you visit, building a personal knowledge graph of the site, all stored locally and encrypted.

View file

@ -0,0 +1,32 @@
# 007: Journey-aware conversation persistence across pages
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** medium
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
Currently each page URL gets its own isolated conversation history. When a user navigates to a new page, their previous conversation is gone. The model should be able to reference earlier conversations from the same browsing session.
## Goal
Allow the model to optionally carry forward conversation context when the user navigates between pages on the same site. Not full history transfer (too expensive), but a compressed summary of what was discussed.
## Architecture
1. When the user leaves a page (or on conversation save), generate a 1-2 sentence conversation summary
2. Store summaries per-page in the site journey data (already tracked)
3. Include recent conversation summaries in the system prompt for new pages
4. Budget: allocate ~500 tokens max for cross-page conversation context
## Example
User on page A asks "What does this function do?" and gets an explanation.
User navigates to page B. The model knows: "On the previous page, you were asking about the fetchData function and how it handles errors."
## Dependencies
- Site journey tracking (implemented)
- Reverse RAG page knowledge (#006)

View file

@ -0,0 +1,29 @@
# 008: Site knowledge graph from browsing patterns
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** low
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
As the user browses a site, build a lightweight knowledge graph of the site's structure: how pages link to each other, what topics each page covers, and what the user has explored vs. what they haven't.
## Goal
The model should understand the site's topology, not just the current page. It can suggest related pages, note when the user has visited related content, and act as a true site guide.
## Architecture
1. Extract internal links from each visited page
2. Map page topics (from page-intelligence.js computed metrics)
3. Build a link graph: `{ page_url: { title, topics, links_to: [urls], visited: bool } }`
4. Store encrypted per-domain
5. Include relevant graph context in system prompt (pages that link to/from current page)
## Dependencies
- Reverse RAG (#006)
- Journey tracking (implemented)
- Page intelligence (implemented)

View file

@ -0,0 +1,38 @@
# 009: Smart context backfill from indexed pages
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** medium
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
When the user opens the chatbot on a page and there is room left in the context window, backfill with content from previously indexed pages on the same site. "Indexed" means any page where the chatbot was opened and page intelligence was computed.
## Goal
Use spare context budget to pull in relevant cross-page knowledge automatically. If the user asks a question about something on a different page they already visited, the model can answer without the user navigating back.
## Architecture
1. After computing page content and journey context, calculate remaining context budget
2. If remaining budget > 2000 tokens, pull from the page knowledge store (#006)
3. Rank stored pages by relevance (keyword overlap with current page, recency)
4. Pack as many page summaries as fit within the remaining budget
5. Include in system prompt as "INDEXED SITE KNOWLEDGE" section
## Budget Calculation
```
total = model max tokens (e.g. 82000)
used = system prompt + page content + journey context + overhead
spare = total - used
backfill = min(spare * 0.5, 10000) // never use more than half of spare
```
## Dependencies
- Reverse RAG page knowledge (#006)
- Journey tracking (implemented)
- fitPageContent() budget system (implemented)

View file

@ -0,0 +1,45 @@
# 010: Ethical crawl mode for site indexing ($14/m plan)
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** medium
**Status:** open
**Affects:** Browser extension (paid tier)
## Description
Paid feature ($14/m) that crawls a site following robots.txt rules, indexing pages for keyword search and cross-page question answering. All data stored in encrypted localStorage on the user's device.
## Goal
Users can index an entire site (or section) without manually opening the chatbot on every page. Then ask questions across all indexed pages like "which page talks about authentication?" or "find all mentions of pricing."
## Architecture
1. User triggers crawl from the chatbot UI (new "Index Site" button in settings)
2. Extension fetches robots.txt, parses disallow/allow rules
3. Starts from current page, follows internal links breadth-first
4. For each page: extract content, compute page intelligence, store summary
5. All data encrypted in vault localStorage
6. Respect: rate limiting (1 req/sec), max depth, max pages, robots.txt
7. Progress UI shows pages indexed, estimated coverage
## Ethics
- Strictly follows robots.txt (no crawl if disallowed)
- Rate limited to 1 request per second minimum
- User-initiated only (never auto-crawl)
- All data stays on the user's device
- Clear disclosure: "This indexes pages for your private use only"
## Monetization
- Free tier: manual page-by-page indexing (open chatbot on each page)
- $14/m plan: automated crawl mode + higher page limits + priority support
- Payment via unsandbox.com accounts
## Dependencies
- Reverse RAG page knowledge (#006)
- Vault encryption (implemented)
- unsandbox.com billing integration

View file

@ -0,0 +1,54 @@
# 011: Vault viewer: indexed pages, charts, and engagement metrics
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** medium
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
A new UI panel in the chatbot showing what's stored in the user's encrypted vault for this site. Charts, metrics, and visualizations that make the user feel like they're getting value just from browsing with the chatbot open.
## Goal
Show the user their browsing intelligence: what pages they've visited, what's been indexed, coverage metrics, and natural engagement data. Not gamified, but genuinely useful metrics that emerge from organic browsing.
## Features
### Pages View
- List of all indexed pages on this domain
- Title, URL, last visited, number of conversations
- Search/filter across indexed pages
- Click to navigate to a page
### Coverage Metrics (natural, not gamified)
- Pages visited vs total internal links discovered
- Topics covered (from page-intelligence.js topic extraction)
- Content consumed: total word count read, estimated reading time
- Questions asked and answered per topic area
- How much of the site's content has been discussed
### Charts
- Timeline of browsing activity on this site
- Topic distribution (pie/bar chart of content categories)
- Engagement depth (pages visited over time)
- Knowledge growth curve (indexed content accumulation)
### Privacy Controls
- Clear indexed data for this site
- Export vault data (encrypted JSON download)
- View storage usage
## Architecture
- New panel accessible from chatbot settings/menu
- Reads from vault localStorage (journey data, page knowledge)
- Charts rendered with lightweight library (no heavy dependencies)
- All computation client-side, no server calls
## Dependencies
- Reverse RAG (#006)
- Journey tracking (implemented)
- Page intelligence (implemented)

View file

@ -0,0 +1,52 @@
# 012: Opt-in chat sharing via unfirehose
**Reporter:** fxhp
**Date:** 2026-03-03
**Priority:** low
**Status:** open
**Affects:** Browser extension, embedded widget
## Description
Allow users to optionally share their chat conversations publicly via unfirehose.org for machine learning training data. All chats are private by default. Sharing is always opt-in, per-conversation, with clear disclosure.
## Goal
Build the ethical pipeline for users to contribute their browsing intelligence and conversations to the public commons, creating training data that improves the models for everyone. Following the unfirehose philosophy: "Pay for privacy, not for access."
## Architecture
1. Per-conversation "Share" button in chat UI
2. On share: decrypt conversation from vault, push to unfirehose.org API
3. Access levels (from unfirehose):
- **public**: Visible in global feed, permanent archive, free
- **unlisted**: Direct link only, not in feeds, paid
- **private**: Default, never leaves device
4. Auth via unsandbox.com SSO (shared identity with unfirehose)
5. API endpoint: `POST https://api.unfirehose.org/v1/ingest`
6. Data format: JSONL with X-Source: "uncloseai" header
## Privacy
- **All chats private by default.** Period.
- Sharing is per-conversation, not global toggle
- Clear preview of what will be shared before sending
- Revocation: user can delete shared conversations
- No metadata leakage: strip device info, IP, vault keys
- GDPR compliant: right to deletion, data portability
## User Flow
1. User has a conversation they want to share
2. Clicks "Share" button on conversation
3. Sees preview: "This will make this conversation publicly visible"
4. Chooses access level (public/unlisted)
5. Authenticates with unsandbox.com account
6. Conversation pushed to unfirehose.org
7. Gets shareable link back
## Dependencies
- unfirehose.org API (see ~/git/unfirehose)
- unsandbox.com accounts/billing
- Vault encryption (implemented)

View file

@ -2,14 +2,22 @@
## Open
| ID | Title | Reporter | Date | Priority |
|----|-------|----------|------|----------|
| 001 | [Extension fails to load on certain pages](001-extension-fails-to-load.md) | cthegray | 2026-03-03 | high | Root cause: CSP blocks script injection. Fix in browser-toys repo. |
| 002 | [TTS stops after first sentence](002-tts-stops-after-first-sentence.md) | cthegray | 2026-03-03 | high | Deferred: TTS disabled via feature flag |
| 003 | [Audio icon freezes entire window](003-audio-icon-freezes-window.md) | cthegray | 2026-03-03 | critical | Deferred: TTS disabled via feature flag |
| 004 | [Add working/spinning indicators for async operations](004-working-indicators.md) | cthegray | 2026-03-03 | medium | CSS ready (.working class). JS wiring when TTS re-enabled. |
| 005 | [Context overflow causes blank or hallucinated responses](005-context-overflow-silent-failure.md) | cthegray | 2026-03-03 | high | Fixed: smart content collapsing in fitPageContent() |
| ID | Title | Priority | Status |
|----|-------|----------|--------|
| 002 | [TTS stops after first sentence](002-tts-stops-after-first-sentence.md) | high | deferred (TTS feature-flagged off) |
| 003 | [Audio icon freezes entire window](003-audio-icon-freezes-window.md) | critical | deferred (TTS feature-flagged off) |
## Closed
## Fixed
None yet.
| ID | Title | Priority | Fix |
|----|-------|----------|-----|
| 001 | [Extension fails to load (CSP)](001-extension-fails-to-load.md) | high | Three-tier injection via chrome.scripting.executeScript |
| 004 | [Working/spinning indicators](004-working-indicators.md) | medium | .working class wired to send button during async ops |
| 005 | [Context overflow silent failure](005-context-overflow-silent-failure.md) | high | Smart content collapsing in fitPageContent() |
| 006 | [Reverse RAG: cross-page context](006-reverse-rag-cross-page-context.md) | medium | Page summaries in encrypted localStorage, backfilled into prompts |
| 007 | [Journey-aware conversation persistence](007-journey-aware-conversation-persistence.md) | medium | Conversation summaries stored in journey data per-page |
| 008 | [Site knowledge graph](008-site-knowledge-graph.md) | low | Internal link extraction + adjacency graph in vault |
| 009 | [Smart context backfill](009-smart-context-backfill.md) | medium | Spare context budget fills with indexed page summaries |
| 010 | [Ethical crawl mode](010-ethical-crawl-mode.md) | medium | robots.txt parser, BFS crawler, rate-limited, API key gated |
| 011 | [Vault viewer dashboard](011-vault-viewer-dashboard.md) | medium | Site intelligence panel with metrics, topics, indexed pages list |
| 012 | [Unfirehose opt-in sharing](012-unfirehose-opt-in-sharing.md) | low | Share button with access level selector, JSONL ingest to unfirehose.org |