Operation Undigg: multi-syntax input, pandoc export, wiki mode, auto-themes.

Phase 1: Pandoc export pipeline — 67 output formats for threads, namespaces, nodes.
Phase 2: Multi-syntax input — accept markdown, HTML, RST, MediaWiki, LaTeX, etc.
Phase 3: Wiki mode — per-namespace toggle, revision tracking, wiki-edit endpoint.
Phase 4: Auto-generated themes — deterministic CSS per namespace, light + dark mode.

Includes unit, integration, and functional tests (95 new, 544 total).
This commit is contained in:
russell@unturf.com 2026-03-10 00:35:51 -04:00
parent f45c42d343
commit 6d1cfff91d
23 changed files with 2822 additions and 14 deletions

120
docs/tickets/15.md Normal file
View file

@ -0,0 +1,120 @@
# T15: Operation Undigg — Pandoc Export & Wiki Mode
**Status**: in-progress
**Priority**: high
**Source**: fox directive 2026-03-09
## Summary
Transform remarkbox into a document-first platform. Every namespace is a book,
every root thread is a chapter. Pandoc renders every format it can produce.
Wiki mode lets anyone edit root topics with revision tracking.
## Architecture
```
Namespace (book) → pandoc → all formats (default)
└── Root Node (chapter) → pandoc → all formats (default)
└── Reply (section) → pandoc → on-demand only
```
### Data model changes
1. **Node.source_format**`Unicode(16)`, default `'markdown'`. Tracks what
syntax the user wrote in (markdown, html, rst, mediawiki, latex, textile, etc).
Pandoc input formats map directly.
2. **rb_revision** — new table for wiki mode edit history.
- `id` (UUIDType, PK)
- `node_id` (FK → Node)
- `user_id` (FK → User)
- `data` (UnicodeText) — raw source at time of edit
- `source_format` (Unicode(16))
- `created` (BigInteger) — timestamp
- `revision_number` (Integer) — sequential per node
3. **Namespace.wiki** — already exists (`Boolean, default=False`), needs
implementation. When True, any authenticated user can edit the root node
of any thread. Each edit creates a revision.
### Export pipeline
Pandoc subprocess. Input = markdown/rst/html/whatever `source_format` says.
Output = every format pandoc supports.
**Default generated (cached on write/edit):**
- Namespace level: one document per namespace containing all root threads
- Root node level: one document per root thread
**On-demand (generated per request):**
- Any node or subthread at any depth
**Pandoc output formats** (67 total, subset for default generation):
- markdown, gfm, commonmark, html5, pdf, epub, docx, odt, rst,
mediawiki, latex, man, plain, rtf, asciidoc, textile, org, json
**URI scheme:**
```
/api/v1/export/namespace/{name}.{format} — full book
/api/v1/export/threads/{node_id}.{format} — single chapter
/api/v1/export/nodes/{node_id}.{format} — on-demand subthread
```
### Content ingestion
On write (create thread, reply, edit):
1. Accept `source_format` parameter (default: `markdown`)
2. If HTML input, strip to clean body (no head/script/style)
3. Store raw source in `Node.data` with `Node.source_format`
4. Render to HTML via pandoc: `pandoc -f {source_format} -t html5`
5. Sanitize HTML output through existing bleach pipeline
6. Store in `Node.data_html`
### Wiki mode
When `Namespace.wiki = True`:
- Any authenticated user can edit root nodes (not just owner/moderator)
- Each edit stores a revision in `rb_revision` before overwriting
- Root node always has the latest version (fast render)
- Revision history accessible via API
- Diff between revisions on-demand
### Progressive enhancement
Export menus work without JS (plain links to format URIs).
With JS: dropdown/popover with format picker, async download.
## Phases
### Phase 1: Export pipeline (pandoc integration)
- [ ] Tree-to-markdown renderer (walk node tree → single markdown document)
- [ ] Export API endpoints (namespace, thread, node)
- [ ] Pandoc subprocess wrapper
- [ ] Format negotiation (URI suffix or Accept header)
### Phase 2: Multi-syntax input
- [ ] Add `source_format` column to Node
- [ ] Alembic migration
- [ ] Modify `set_data()` to use pandoc for non-markdown formats
- [ ] Accept `source_format` on create/reply/edit API endpoints
- [ ] HTML stripping for HTML input
### Phase 3: Wiki mode + revisions
- [ ] Create `rb_revision` model + migration
- [ ] Implement wiki edit permissions in Namespace
- [ ] Store revisions on edit
- [ ] Revision history API endpoint
- [ ] Diff endpoint
### Phase 4: Auto-generated themes
- [ ] Per-namespace theme generation (light + dark)
- [ ] CSS custom properties for theming
- [ ] Theme preview
## Notes
- Pandoc 3.1.3 installed at `/usr/bin/pandoc`
- 43 input formats, 67 output formats
- PDF requires LaTeX (`pdflatex`) or `wkhtmltopdf` — check availability
- `Namespace.wiki` column already exists in schema, just needs implementation
- Export cache invalidation: on any node edit in the tree

View file

@ -19,3 +19,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02-
| [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` |
| [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` |
| [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 |
| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | in-progress | high | fox directive 2026-03-09 |