From 93926bfbfc3bea03d7238921d83e21b1a1970d9b Mon Sep 17 00:00:00 2001
From: "russell@unturf.com" <russell@unturf.com>
Date: Tue, 3 Mar 2026 16:26:01 -0500
Subject: [PATCH] smart content collapsing for context overflow, spinner CSS,
 ticket updates

fitPageContent() in content.js: 6 progressive collapse stages (dedup,
nav links, references, lists, code blocks, sidebar debris) before hard
truncation. No summarization, real content structurally compressed.
Wired into both page extraction points in embed modal.
Added .working pulse and .uncloseai-spinner CSS animations for #004.
Updated tickets #001 (CSP root cause), #004 (CSS ready), #005 (fixed).
---
 docs/tickets/001-extension-fails-to-load.md   |  12 +-
 docs/tickets/004-working-indicators.md        |  13 +-
 .../005-context-overflow-silent-failure.md    |  16 +-
 docs/tickets/index.md                         |  10 +-
 public/src/content.js                         | 210 ++++++++++++++++++
 public/src/uncloseai-embed-modal.js           |  14 +-
 public/src/uncloseai-modal-builtin.css        |  26 +++
 public/src/uncloseai-modal-pico.css           |  26 +++
 8 files changed, 315 insertions(+), 12 deletions(-)

diff --git a/docs/tickets/001-extension-fails-to-load.md b/docs/tickets/001-extension-fails-to-load.md
index 41ef406..09149e0 100644
--- a/docs/tickets/001-extension-fails-to-load.md
+++ b/docs/tickets/001-extension-fails-to-load.md
@@ -30,6 +30,16 @@ Possible causes:
 - Shadow DOM or aggressive DOM manipulation interfering with injection
 - Extension manifest permissions not covering these domains
 
+## Root Cause
+
+The browser-toys extension injects `<script src="https://uncloseai.com/uncloseai.js" type="module">` via DOM. Pages with strict CSP `script-src` directives block this because `https://uncloseai.com` is not in their allowlist.
+
+## Fix
+
+In the `uncloseai-browser-toys` repo, switch the content script from DOM script injection to `chrome.scripting.executeScript()` with the bundled IIFE (`uncloseai-bundle.js`). Extensions executing their own bundled code bypass page CSP entirely.
+
+The bundle is ready: `make bundle-extension` produces `public/uncloseai-bundle.js` (1.6MB IIFE, CSP-safe, no ES modules, no CDN imports).
+
 ## Notes
 
-Need to inspect the extension's content script injection mechanism and compare CSP headers between working and non-working pages.
+Fix lives in the browser-toys repo, not this one. The bundle and architecture are ready here.
diff --git a/docs/tickets/004-working-indicators.md b/docs/tickets/004-working-indicators.md
index ec5bea3..7b40b34 100644
--- a/docs/tickets/004-working-indicators.md
+++ b/docs/tickets/004-working-indicators.md
@@ -28,6 +28,17 @@ Keep consistent with the existing hourglass pattern on the book icon. Options:
 - Icon swap to an animated version during loading
 - Pulsing opacity animation (subtle but effective)
 
+## Implementation
+
+CSS animations added to both theme files (`uncloseai-modal-builtin.css`, `uncloseai-modal-pico.css`):
+- `.working` class: pulse animation (opacity 1 to 0.4), disables pointer events
+- `.uncloseai-spinner` class: continuous rotation for inline spinner elements
+- Apply to `.uncloseai-btn-small` or `.uncloseai-btn-primary` elements
+
+Usage: `btn.classList.add("working")` when async starts, `.remove("working")` when done.
+
+TTS buttons (primary consumers) are currently hidden via feature flag. When TTS is re-enabled, wire the `.working` class into the TTS button click handlers.
+
 ## Notes
 
-This is a UX quality issue. Without feedback, users assume the tap didn't register and tap again, potentially triggering duplicate requests or the freeze described in ticket 003.
+CSS is ready. JS wiring deferred until TTS is re-enabled (tickets #002, #003).
diff --git a/docs/tickets/005-context-overflow-silent-failure.md b/docs/tickets/005-context-overflow-silent-failure.md
index 08ac8e7..d8a2c80 100644
--- a/docs/tickets/005-context-overflow-silent-failure.md
+++ b/docs/tickets/005-context-overflow-silent-failure.md
@@ -33,6 +33,20 @@ Possible fixes:
 - Never silently swallow API errors: if the completion fails, show the error to the user
 - Prioritize user message and page content over system prompt when truncating
 
+## Fix Implemented
+
+Smart content collapsing in `content.js` via `fitPageContent()`. Progressive stages run in order of aggressiveness, stopping when content fits:
+
+1. **Deduplicate paragraphs**: remove exact duplicate blocks (common in scraped DOM)
+2. **Collapse nav links**: 5+ consecutive links become `[Navigation: N links including ...]`
+3. **Collapse references**: dense URL/citation clusters become `[... N references collapsed]`
+4. **Collapse lists**: 7+ item lists keep first 5, note `[... N more items]`
+5. **Collapse code blocks**: 12+ line blocks keep first 10, note `[... N more lines]`
+6. **Collapse sidebar debris**: runs of 6+ short fragments get compressed
+7. **Hard truncate**: last resort, cuts at paragraph/sentence boundary
+
+No summarization. All content stays real, just structurally compressed. Uses `max_model_len` from model discovery with 15k token overhead reserved for system prompt, computed intel, chat history, and output.
+
 ## Notes
 
-The `max_model_len` field from model discovery should be used to calculate available context budget. System prompt + conversation history + page content must fit within this limit. If it doesn't, the client needs a truncation strategy, not silent failure.
+Hermes bumped to 82k context. Wikipedia Napoleon (~67k tokens) should fit after collapsing without hard truncation.
diff --git a/docs/tickets/index.md b/docs/tickets/index.md
index a5ada4c..fc2875c 100644
--- a/docs/tickets/index.md
+++ b/docs/tickets/index.md
@@ -4,11 +4,11 @@
 
 | ID | Title | Reporter | Date | Priority |
 |----|-------|----------|------|----------|
-| 001 | [Extension fails to load on certain pages](001-extension-fails-to-load.md) | cthegray | 2026-03-03 | high |
-| 002 | [TTS stops after first sentence](002-tts-stops-after-first-sentence.md) | cthegray | 2026-03-03 | high |
-| 003 | [Audio icon freezes entire window](003-audio-icon-freezes-window.md) | cthegray | 2026-03-03 | critical |
-| 004 | [Add working/spinning indicators for async operations](004-working-indicators.md) | cthegray | 2026-03-03 | medium |
-| 005 | [Context overflow causes blank or hallucinated responses](005-context-overflow-silent-failure.md) | cthegray | 2026-03-03 | high |
+| 001 | [Extension fails to load on certain pages](001-extension-fails-to-load.md) | cthegray | 2026-03-03 | high | Root cause: CSP blocks script injection. Fix in browser-toys repo. |
+| 002 | [TTS stops after first sentence](002-tts-stops-after-first-sentence.md) | cthegray | 2026-03-03 | high | Deferred: TTS disabled via feature flag |
+| 003 | [Audio icon freezes entire window](003-audio-icon-freezes-window.md) | cthegray | 2026-03-03 | critical | Deferred: TTS disabled via feature flag |
+| 004 | [Add working/spinning indicators for async operations](004-working-indicators.md) | cthegray | 2026-03-03 | medium | CSS ready (.working class). JS wiring when TTS re-enabled. |
+| 005 | [Context overflow causes blank or hallucinated responses](005-context-overflow-silent-failure.md) | cthegray | 2026-03-03 | high | Fixed: smart content collapsing in fitPageContent() |
 
 ## Closed
 
diff --git a/public/src/content.js b/public/src/content.js
index 068219a..e74f2b1 100644
--- a/public/src/content.js
+++ b/public/src/content.js
@@ -151,6 +151,216 @@ export async function extractWebpageContent() {
 	return extractDOMContent();
 }
 
+// Estimate tokens: ~4 chars per token for English prose
+function estimateTokens(text) {
+	return Math.ceil(text.length / 4);
+}
+
+// Collapse navigation link clusters: 5+ consecutive links become a single line
+function collapseNavLinks(text) {
+	// Match runs of lines that are just markdown links, possibly with separators
+	return text.replace(
+		/(\[([^\]]*)\]\([^)]*\)\s*){5,}/g,
+		(match) => {
+			const links = match.match(/\[([^\]]*)\]\([^)]*\)/g);
+			if (!links) return match;
+			const first3 = links.slice(0, 3).map(l => l.match(/\[([^\]]*)\]/)?.[1]).filter(Boolean);
+			return `[Navigation: ${links.length} links including ${first3.join(", ")}]\n`;
+		},
+	);
+}
+
+// Collapse long lists: keep first 5 items, note the rest
+function collapseLists(text) {
+	const lines = text.split("\n");
+	const result = [];
+	let listRun = [];
+	let inList = false;
+
+	const flushList = () => {
+		if (listRun.length <= 7) {
+			result.push(...listRun);
+		} else {
+			result.push(...listRun.slice(0, 5));
+			result.push(`  [... ${listRun.length - 5} more items]`);
+		}
+		listRun = [];
+		inList = false;
+	};
+
+	for (const line of lines) {
+		const isList = /^\s*[-*•]\s/.test(line) || /^\s*\d+[.)]\s/.test(line);
+		if (isList) {
+			inList = true;
+			listRun.push(line);
+		} else {
+			if (inList) flushList();
+			result.push(line);
+		}
+	}
+	if (inList) flushList();
+	return result.join("\n");
+}
+
+// Collapse code blocks: keep first 10 lines, note the rest
+function collapseCodeBlocks(text) {
+	return text.replace(/```[\w]*\n([\s\S]*?)```/g, (match, code) => {
+		const lines = code.split("\n");
+		if (lines.length <= 12) return match;
+		const lang = match.match(/```(\w*)/)?.[1] || "";
+		return `\`\`\`${lang}\n${lines.slice(0, 10).join("\n")}\n[... ${lines.length - 10} more lines]\n\`\`\``;
+	});
+}
+
+// Remove exact duplicate paragraphs (common in scraped DOM content)
+function deduplicateParagraphs(text) {
+	const paragraphs = text.split("\n\n");
+	const seen = new Set();
+	const result = [];
+	for (const p of paragraphs) {
+		const trimmed = p.trim();
+		if (!trimmed) continue;
+		// Normalize whitespace for comparison
+		const key = trimmed.replace(/\s+/g, " ");
+		if (key.length > 20 && seen.has(key)) continue;
+		seen.add(key);
+		result.push(p);
+	}
+	return result.join("\n\n");
+}
+
+// Collapse reference/citation sections (Wikipedia pattern: dense URL clusters)
+function collapseReferences(text) {
+	const lines = text.split("\n");
+	const result = [];
+	let refRun = [];
+	let inRefs = false;
+
+	const flushRefs = () => {
+		if (refRun.length <= 5) {
+			result.push(...refRun);
+		} else {
+			result.push(refRun[0]);
+			result.push(`[... ${refRun.length - 1} references collapsed]`);
+		}
+		refRun = [];
+		inRefs = false;
+	};
+
+	for (const line of lines) {
+		// Lines that are mostly URLs or citation markers
+		const urlCount = (line.match(/https?:\/\//g) || []).length;
+		const isRef = urlCount >= 2 || (urlCount >= 1 && line.length < 200 && /\[\d+\]|\^/.test(line));
+		if (isRef) {
+			inRefs = true;
+			refRun.push(line);
+		} else {
+			if (inRefs) flushRefs();
+			result.push(line);
+		}
+	}
+	if (inRefs) flushRefs();
+	return result.join("\n");
+}
+
+// Collapse consecutive short lines (sidebar fragments, metadata debris)
+function collapseSidebarDebris(text) {
+	const lines = text.split("\n");
+	const result = [];
+	let shortRun = [];
+
+	const flushShort = () => {
+		if (shortRun.length <= 6) {
+			result.push(...shortRun);
+		} else {
+			result.push(...shortRun.slice(0, 3));
+			result.push(`[... ${shortRun.length - 3} short fragments collapsed]`);
+		}
+		shortRun = [];
+	};
+
+	for (const line of lines) {
+		// Short lines with little content (under 30 chars, not headings or metadata)
+		const trimmed = line.trim();
+		if (trimmed.length > 0 && trimmed.length < 30 && !/^\*\*|^#|^---/.test(trimmed)) {
+			shortRun.push(line);
+		} else {
+			if (shortRun.length > 6) flushShort();
+			else if (shortRun.length > 0) { result.push(...shortRun); shortRun = []; }
+			result.push(line);
+		}
+	}
+	if (shortRun.length > 6) flushShort();
+	else result.push(...shortRun);
+	return result.join("\n");
+}
+
+// Fit page content to model context using progressive structural collapsing.
+// No summarization: content is real, just structurally compressed.
+// Stages run in order of aggressiveness, stopping when content fits.
+export function fitPageContent(pageContent, maxModelTokens) {
+	// Reserve tokens for everything except page content:
+	// system prompt ~300, computed intel ~200, analysis ~300,
+	// wrapper text ~100, chat history ~8000, output ~4000, safety ~2000
+	const overhead = 15000;
+	const maxPageTokens = Math.max(0, maxModelTokens - overhead);
+	const originalTokens = estimateTokens(pageContent);
+
+	if (originalTokens <= maxPageTokens) {
+		return { content: pageContent, truncated: false, originalTokens, finalTokens: originalTokens };
+	}
+
+	console.log(`Page content (~${originalTokens.toLocaleString()} tokens) exceeds budget (~${maxPageTokens.toLocaleString()} tokens), collapsing...`);
+
+	// Progressive collapse stages, least aggressive first
+	const stages = [
+		{ name: "dedup paragraphs", fn: deduplicateParagraphs },
+		{ name: "collapse nav links", fn: collapseNavLinks },
+		{ name: "collapse references", fn: collapseReferences },
+		{ name: "collapse lists", fn: collapseLists },
+		{ name: "collapse code blocks", fn: collapseCodeBlocks },
+		{ name: "collapse sidebar debris", fn: collapseSidebarDebris },
+	];
+
+	let content = pageContent;
+	let appliedStages = [];
+
+	for (const stage of stages) {
+		content = stage.fn(content);
+		const tokens = estimateTokens(content);
+		appliedStages.push(stage.name);
+		console.log(`  ${stage.name}: ~${tokens.toLocaleString()} tokens`);
+		if (tokens <= maxPageTokens) {
+			const finalTokens = estimateTokens(content);
+			console.log(`Content fits after ${appliedStages.length} collapse stages: ~${originalTokens.toLocaleString()} → ~${finalTokens.toLocaleString()} tokens`);
+			return { content, truncated: true, originalTokens, finalTokens, stages: appliedStages };
+		}
+	}
+
+	// All stages applied, still too large: hard truncate at paragraph boundary
+	const maxChars = maxPageTokens * 4;
+	const slice = content.substring(0, maxChars);
+	const lastParagraph = slice.lastIndexOf("\n\n");
+	const lastSentence = slice.lastIndexOf(". ");
+	const breakPoint =
+		lastParagraph > maxChars * 0.8 ? lastParagraph :
+		lastSentence > maxChars * 0.8 ? lastSentence + 1 :
+		maxChars;
+
+	const finalContent = content.substring(0, breakPoint);
+	const finalTokens = estimateTokens(finalContent);
+
+	console.warn(`Page content hard truncated after all collapse stages: ~${originalTokens.toLocaleString()} → ~${finalTokens.toLocaleString()} tokens`);
+
+	return {
+		content: `${finalContent}\n\n[Content collapsed and truncated: ~${originalTokens.toLocaleString()} tokens → ~${finalTokens.toLocaleString()} tokens]`,
+		truncated: true,
+		originalTokens,
+		finalTokens,
+		stages: [...appliedStages, "hard truncate"],
+	};
+}
+
 // --- Page Analysis (pre-inference classification) ---
 // Runs before the greeting to extract structured intelligence about the page.
 // The greeting uses these keywords instead of guessing from raw text.
diff --git a/public/src/uncloseai-embed-modal.js b/public/src/uncloseai-embed-modal.js
index c1dc41c..2cc9744 100644
--- a/public/src/uncloseai-embed-modal.js
+++ b/public/src/uncloseai-embed-modal.js
@@ -18,7 +18,7 @@
 import { marked } from "https://cdn.jsdelivr.net/npm/marked/lib/marked.esm.js";
 import hljs from "https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.10.0/es/highlight.min.js";
 import { setSystemMessageAppend, isTTSEnabled } from "./config.js";
-import { extractWebpageContent, buildPageAnalysisPrompt, parsePageAnalysis, buildAnalysisContext } from "./content.js";
+import { extractWebpageContent, fitPageContent, buildPageAnalysisPrompt, parsePageAnalysis, buildAnalysisContext } from "./content.js";
 import { computePageIntelligence, formatPageIntelligence } from "./page-intelligence.js";
 import {
 	detectCurrentTheme,
@@ -2050,7 +2050,10 @@ async function openUncloseaiEmbeddedModalNew() {
 				console.log("History length:", history ? history.length : 0);
 				if (history && history.length > 0) {
 					// Set up page context for existing conversation
-					const pageContent = await extractWebpageContent();
+					const { getSelectedModelMaxTokens } = await import("./models.js");
+					const rawPageContent = await extractWebpageContent();
+					const fitted = fitPageContent(rawPageContent, getSelectedModelMaxTokens());
+					const pageContent = fitted.content;
 					const pageTitle = document.title || window.location.hostname;
 
 					// Computed intelligence: instant, no API call, free for returning users
@@ -2488,8 +2491,11 @@ You have complete knowledge of this page content and can reference any details,
 			chatBox.appendChild(introMsg);
 
 			try {
-				// Get page content for context
-				const pageContent = await extractWebpageContent();
+				// Get page content for context, truncate to fit model context window
+				const { getSelectedModelMaxTokens } = await import("./models.js");
+				const rawPageContent = await extractWebpageContent();
+				const fitted = fitPageContent(rawPageContent, getSelectedModelMaxTokens());
+				const pageContent = fitted.content;
 				const pageTitle = document.title || window.location.hostname;
 
 				// COMPUTED INTELLIGENCE: instant, deterministic, zero API calls
diff --git a/public/src/uncloseai-modal-builtin.css b/public/src/uncloseai-modal-builtin.css
index 14b77a3..b71c75f 100644
--- a/public/src/uncloseai-modal-builtin.css
+++ b/public/src/uncloseai-modal-builtin.css
@@ -1356,6 +1356,32 @@ dialog#uncloseai-embedded-modal[data-theme="dark"] .user-message {
   cursor: wait;
 }
 
+/* Spinner animation for async operation buttons (ticket #004) */
+@keyframes uncloseai-spin {
+  from { transform: rotate(0deg); }
+  to { transform: rotate(360deg); }
+}
+
+@keyframes uncloseai-pulse {
+  0%, 100% { opacity: 1; }
+  50% { opacity: 0.4; }
+}
+
+.uncloseai-btn-small.working {
+  pointer-events: none;
+  animation: uncloseai-pulse 1.2s ease-in-out infinite;
+}
+
+.uncloseai-btn-primary.working {
+  pointer-events: none;
+  animation: uncloseai-pulse 1.2s ease-in-out infinite;
+}
+
+.uncloseai-spinner {
+  display: inline-block;
+  animation: uncloseai-spin 1s linear infinite;
+}
+
 /* =================================================================== */
 /* WIDGET LIBRARY STYLES */
 /* =================================================================== */
diff --git a/public/src/uncloseai-modal-pico.css b/public/src/uncloseai-modal-pico.css
index a41dbb4..7a00801 100644
--- a/public/src/uncloseai-modal-pico.css
+++ b/public/src/uncloseai-modal-pico.css
@@ -1434,6 +1434,32 @@ dialog#uncloseai-embedded-modal[data-theme="dark"] .user-message {
   cursor: wait;
 }
 
+/* Spinner animation for async operation buttons (ticket #004) */
+@keyframes uncloseai-spin {
+  from { transform: rotate(0deg); }
+  to { transform: rotate(360deg); }
+}
+
+@keyframes uncloseai-pulse {
+  0%, 100% { opacity: 1; }
+  50% { opacity: 0.4; }
+}
+
+.uncloseai-btn-small.working {
+  pointer-events: none;
+  animation: uncloseai-pulse 1.2s ease-in-out infinite;
+}
+
+.uncloseai-btn-primary.working {
+  pointer-events: none;
+  animation: uncloseai-pulse 1.2s ease-in-out infinite;
+}
+
+.uncloseai-spinner {
+  display: inline-block;
+  animation: uncloseai-spin 1s linear infinite;
+}
+
 /* =================================================================== */
 /* WIDGET LIBRARY STYLES - PICOCSS OVERRIDES */
 /* =================================================================== */