Operation Undigg: multi-syntax input, pandoc export, wiki mode, auto-themes.
Phase 1: Pandoc export pipeline — 67 output formats for threads, namespaces, nodes. Phase 2: Multi-syntax input — accept markdown, HTML, RST, MediaWiki, LaTeX, etc. Phase 3: Wiki mode — per-namespace toggle, revision tracking, wiki-edit endpoint. Phase 4: Auto-generated themes — deterministic CSS per namespace, light + dark mode. Includes unit, integration, and functional tests (95 new, 544 total).
This commit is contained in:
parent
f45c42d343
commit
6d1cfff91d
23 changed files with 2822 additions and 14 deletions
120
docs/tickets/15.md
Normal file
120
docs/tickets/15.md
Normal file
|
|
@ -0,0 +1,120 @@
|
|||
# T15: Operation Undigg — Pandoc Export & Wiki Mode
|
||||
|
||||
**Status**: in-progress
|
||||
**Priority**: high
|
||||
**Source**: fox directive 2026-03-09
|
||||
|
||||
## Summary
|
||||
|
||||
Transform remarkbox into a document-first platform. Every namespace is a book,
|
||||
every root thread is a chapter. Pandoc renders every format it can produce.
|
||||
Wiki mode lets anyone edit root topics with revision tracking.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Namespace (book) → pandoc → all formats (default)
|
||||
└── Root Node (chapter) → pandoc → all formats (default)
|
||||
└── Reply (section) → pandoc → on-demand only
|
||||
```
|
||||
|
||||
### Data model changes
|
||||
|
||||
1. **Node.source_format** — `Unicode(16)`, default `'markdown'`. Tracks what
|
||||
syntax the user wrote in (markdown, html, rst, mediawiki, latex, textile, etc).
|
||||
Pandoc input formats map directly.
|
||||
|
||||
2. **rb_revision** — new table for wiki mode edit history.
|
||||
- `id` (UUIDType, PK)
|
||||
- `node_id` (FK → Node)
|
||||
- `user_id` (FK → User)
|
||||
- `data` (UnicodeText) — raw source at time of edit
|
||||
- `source_format` (Unicode(16))
|
||||
- `created` (BigInteger) — timestamp
|
||||
- `revision_number` (Integer) — sequential per node
|
||||
|
||||
3. **Namespace.wiki** — already exists (`Boolean, default=False`), needs
|
||||
implementation. When True, any authenticated user can edit the root node
|
||||
of any thread. Each edit creates a revision.
|
||||
|
||||
### Export pipeline
|
||||
|
||||
Pandoc subprocess. Input = markdown/rst/html/whatever `source_format` says.
|
||||
Output = every format pandoc supports.
|
||||
|
||||
**Default generated (cached on write/edit):**
|
||||
- Namespace level: one document per namespace containing all root threads
|
||||
- Root node level: one document per root thread
|
||||
|
||||
**On-demand (generated per request):**
|
||||
- Any node or subthread at any depth
|
||||
|
||||
**Pandoc output formats** (67 total, subset for default generation):
|
||||
- markdown, gfm, commonmark, html5, pdf, epub, docx, odt, rst,
|
||||
mediawiki, latex, man, plain, rtf, asciidoc, textile, org, json
|
||||
|
||||
**URI scheme:**
|
||||
```
|
||||
/api/v1/export/namespace/{name}.{format} — full book
|
||||
/api/v1/export/threads/{node_id}.{format} — single chapter
|
||||
/api/v1/export/nodes/{node_id}.{format} — on-demand subthread
|
||||
```
|
||||
|
||||
### Content ingestion
|
||||
|
||||
On write (create thread, reply, edit):
|
||||
1. Accept `source_format` parameter (default: `markdown`)
|
||||
2. If HTML input, strip to clean body (no head/script/style)
|
||||
3. Store raw source in `Node.data` with `Node.source_format`
|
||||
4. Render to HTML via pandoc: `pandoc -f {source_format} -t html5`
|
||||
5. Sanitize HTML output through existing bleach pipeline
|
||||
6. Store in `Node.data_html`
|
||||
|
||||
### Wiki mode
|
||||
|
||||
When `Namespace.wiki = True`:
|
||||
- Any authenticated user can edit root nodes (not just owner/moderator)
|
||||
- Each edit stores a revision in `rb_revision` before overwriting
|
||||
- Root node always has the latest version (fast render)
|
||||
- Revision history accessible via API
|
||||
- Diff between revisions on-demand
|
||||
|
||||
### Progressive enhancement
|
||||
|
||||
Export menus work without JS (plain links to format URIs).
|
||||
With JS: dropdown/popover with format picker, async download.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 1: Export pipeline (pandoc integration)
|
||||
- [ ] Tree-to-markdown renderer (walk node tree → single markdown document)
|
||||
- [ ] Export API endpoints (namespace, thread, node)
|
||||
- [ ] Pandoc subprocess wrapper
|
||||
- [ ] Format negotiation (URI suffix or Accept header)
|
||||
|
||||
### Phase 2: Multi-syntax input
|
||||
- [ ] Add `source_format` column to Node
|
||||
- [ ] Alembic migration
|
||||
- [ ] Modify `set_data()` to use pandoc for non-markdown formats
|
||||
- [ ] Accept `source_format` on create/reply/edit API endpoints
|
||||
- [ ] HTML stripping for HTML input
|
||||
|
||||
### Phase 3: Wiki mode + revisions
|
||||
- [ ] Create `rb_revision` model + migration
|
||||
- [ ] Implement wiki edit permissions in Namespace
|
||||
- [ ] Store revisions on edit
|
||||
- [ ] Revision history API endpoint
|
||||
- [ ] Diff endpoint
|
||||
|
||||
### Phase 4: Auto-generated themes
|
||||
- [ ] Per-namespace theme generation (light + dark)
|
||||
- [ ] CSS custom properties for theming
|
||||
- [ ] Theme preview
|
||||
|
||||
## Notes
|
||||
|
||||
- Pandoc 3.1.3 installed at `/usr/bin/pandoc`
|
||||
- 43 input formats, 67 output formats
|
||||
- PDF requires LaTeX (`pdflatex`) or `wkhtmltopdf` — check availability
|
||||
- `Namespace.wiki` column already exists in schema, just needs implementation
|
||||
- Export cache invalidation: on any node edit in the tree
|
||||
|
|
@ -19,3 +19,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02-
|
|||
| [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` |
|
||||
| [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` |
|
||||
| [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 |
|
||||
| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | in-progress | high | fox directive 2026-03-09 |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue