diff --git a/docs/api.md b/docs/api.md index d73d19e..1d0ce8e 100644 --- a/docs/api.md +++ b/docs/api.md @@ -190,6 +190,7 @@ Request body: - `namespace` (required) - `title` (required) - `data` (required, max 500000 chars) +- `source_format` (optional, default `"markdown"`) — any pandoc input format - `anonymous_name` (optional, used when namespace allows anonymous) - `email` (optional, creates an unverified user) @@ -220,6 +221,7 @@ Request body: ``` - `data` (required, max 500000 chars) +- `source_format` (optional, default `"markdown"`) - `anonymous_name` (optional) - `email` (optional) @@ -264,12 +266,14 @@ Request body: ```json { "data": "Updated markdown", - "title": "Updated Title" + "title": "Updated Title", + "source_format": "markdown" } ``` - `data` (optional, updates content) - `title` (optional, only applies to root nodes) +- `source_format` (optional, default `"markdown"`) At least one of `data` or `title` is required. @@ -350,6 +354,214 @@ Error `401`: } ``` +### Multi-Syntax Input + +All write endpoints (`POST /threads`, `POST /replies`, `PATCH /nodes`) accept +an optional `source_format` parameter. Default is `"markdown"`. + +Supported input formats include any pandoc-supported format: `markdown`, `html`, +`rst`, `mediawiki`, `latex`, `textile`, `org`, `docbook`, `commonmark`, etc. + +```json +{ + "data": "Title\n=====\n\nA paragraph in reStructuredText.", + "source_format": "rst" +} +``` + +HTML input is round-tripped through pandoc (html → markdown) to produce a clean +canonical source. All formats are rendered to HTML via pandoc and sanitized +through the bleach pipeline before storage. + +The `source_format` field is included in all node serializations. + +--- + +### Export Formats + +``` +GET /api/v1/export/formats +``` + +Returns all available pandoc output formats. + +Response `200`: +```json +{ + "formats": ["asciidoc", "commonmark", "docx", "epub", "html5", "latex", "markdown", "pdf", "rst", "..."], + "count": 67 +} +``` + +--- + +### Export Thread + +``` +GET /api/v1/export/threads/{node_id}.{format} +``` + +Exports a single thread (root + replies) as a document. + +Examples: +``` +GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.pdf +GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.epub +GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.md +``` + +Binary formats (pdf, epub, docx) return the file with `Content-Disposition: attachment`. +Text formats return inline with appropriate content type. + +--- + +### Export Namespace + +``` +GET /api/v1/export/namespace/{namespace_name}.{format} +``` + +Exports an entire namespace as a book. Each root thread becomes a chapter. + +Examples: +``` +GET /api/v1/export/namespace/meta.remarkbox.com.epub +GET /api/v1/export/namespace/meta.remarkbox.com.pdf +``` + +--- + +### Export Node (On-Demand) + +``` +GET /api/v1/export/nodes/{node_id}.{format} +``` + +Exports any node and its subtree. Useful for exporting a specific subthread +at any nesting depth. + +--- + +### Wiki Edit + +``` +POST /api/v1/nodes/{node_id}/wiki-edit +``` + +Wiki-edit a root node. Creates a revision snapshot before applying the edit. +Requires authentication. The namespace must have `wiki = True`, or the user +must be the node owner/moderator. + +Request body: +```json +{ + "data": "Updated wiki content", + "source_format": "markdown" +} +``` + +Response `200`: +```json +{ + "node": {"id": "...", "data": "Updated wiki content", "...": "..."}, + "revision": {"id": "...", "revision_number": 2, "...": "..."} +} +``` + +Errors: +- `401` if not authenticated +- `403` if wiki editing not allowed for this user/node +- `404` if node not found + +--- + +### Node Revisions + +``` +GET /api/v1/nodes/{node_id}/revisions +``` + +Returns the revision history for a node. + +Response `200`: +```json +{ + "node_id": "...", + "revisions": [ + { + "id": "...", + "revision_number": 1, + "data": "Original content", + "source_format": "markdown", + "created": 1710043200000, + "user": {"id": "...", "name": "timehexon"} + } + ] +} +``` + +--- + +### Get Revision + +``` +GET /api/v1/revisions/{revision_id} +``` + +Returns a specific revision by ID. + +Response `200`: +```json +{ + "revision": { + "id": "...", + "node_id": "...", + "revision_number": 1, + "data": "Content at this revision", + "source_format": "markdown", + "created": 1710043200000, + "user": {"id": "...", "name": "timehexon"} + } +} +``` + +--- + +### Theme CSS + +``` +GET /api/v1/themes/{namespace_name}/css +``` + +Returns auto-generated CSS theme for a namespace. Deterministic — same +namespace always produces the same theme. Includes light mode (`:root`, +`.theme-light`) and dark mode (`@media (prefers-color-scheme: dark)`, +`.theme-dark`). + +Response: `200 text/css` with 1-day cache header. + +--- + +### Theme Preview + +``` +GET /api/v1/themes/{namespace_name}/preview +``` + +Returns the theme color palette as JSON for previewing without loading CSS. + +Response `200`: +```json +{ + "namespace": "meta.remarkbox.com", + "hue": 217, + "light": {"bg": "#f8f9fa", "text": "#1a1a2e", "link": "#2563eb", "...": "..."}, + "dark": {"bg": "#0f0f1a", "text": "#e8e8f0", "link": "#60a5fa", "...": "..."} +} +``` + +--- + ## Error Format All errors return a JSON body with an `error` key: diff --git a/docs/architecture-undigg.md b/docs/architecture-undigg.md new file mode 100644 index 0000000..d7f198a --- /dev/null +++ b/docs/architecture-undigg.md @@ -0,0 +1,134 @@ +# Operation Undigg — Architecture + +Document-first platform: every namespace is a book, every thread is a chapter, +every reply is a section. Pandoc renders 67 output formats. + +## Data Flow + +``` + ┌─────────────────────────────────────────────┐ + │ Write Path │ + │ │ + User Input │ source_format pandoc bleach │ + (md/html/rst/ │ ─────────────► convert to ──► sanitize │ + mediawiki/latex/ │ html5 pipeline │ + textile/org/...) │ │ │ │ + │ │ ▼ │ + │ │ data_html │ + │ │ (rendered) │ + │ │ │ + │ html input: │ │ + │ pandoc html→md │ │ + │ (round-trip clean) │ │ + │ │ │ │ + │ ▼ │ │ + │ data (raw) │ │ + │ + source_format │ │ + └─────────────────────────────────────────────┘ + + ┌─────────────────────────────────────────────┐ + │ Read / Export Path │ + │ │ + Node tree │ tree-to-markdown pandoc │ + (adjacency list) ────► │ renderer ──► convert to ────► output + │ target format │ + │ │ + │ Hierarchy: │ + │ # Thread Title (h1) │ + │ ## Author Name (h2, per reply) │ + │ ### Author Name (h3, nested reply) │ + │ ...depth maps to heading level │ + └─────────────────────────────────────────────┘ +``` + +## Export Hierarchy + +``` + Namespace (book) + GET /api/v1/export/namespace/{name}.{fmt} + │ + ├── Root Node A (chapter) ← default export + │ GET /api/v1/export/threads/{id}.{fmt} + │ │ + │ ├── Reply 1 (section) ← on-demand + │ │ GET /api/v1/export/nodes/{id}.{fmt} + │ │ │ + │ │ └── Reply 1.1 (subsection) ← on-demand + │ │ + │ └── Reply 2 (section) ← on-demand + │ + └── Root Node B (chapter) ← default export + GET /api/v1/export/threads/{id}.{fmt} +``` + +**67 output formats** including: markdown, html5, pdf, epub, docx, odt, rst, +latex, mediawiki, man, plain, rtf, asciidoc, textile, org, json, and more. + +## Wiki Mode + +``` + ┌──────────────┐ wiki_edit() ┌──────────────┐ + │ │ ──────────────────► │ rb_revision │ + │ rb_node │ │ │ + │ │ 1. snapshot current │ id │ + │ id │ data → revision │ node_id (FK) │ + │ data │ 2. increment rev# │ user_id (FK) │ + │ data_html │ 3. overwrite node │ data │ + │ source_fmt │ with new content │ source_format │ + │ │ │ revision_number│ + └──────────────┘ │ created │ + │ └──────────────┘ + │ + ▼ + can_wiki_edit(node, user)? + ├── user.authenticated? → required + ├── namespace.can_alter_node()? → owner/moderator always yes + └── namespace.wiki && node.is_root? → wiki mode for root nodes +``` + +## Auto-Generated Themes + +``` + namespace_name + │ + ▼ + SHA-256 hash + │ + ├── bits[0:8] → hue (0-360) + ├── bits[8:40] → 8 seed values + │ + ▼ + HSL color palette + │ + ├── Light mode (:root, .theme-light) + │ --rb-bg, --rb-text, --rb-link, --rb-accent, --rb-border, ... + │ + └── Dark mode (@media prefers-color-scheme: dark, .theme-dark) + --rb-bg, --rb-text, --rb-link, --rb-accent, --rb-border, ... + + Deterministic: same namespace → same theme, always. + Served at: GET /api/v1/themes/{namespace}/css (1-day cache) + Preview: GET /api/v1/themes/{namespace}/preview (JSON palette) +``` + +## Module Map + +``` + remarkbox/ + ├── lib/ + │ ├── pandoc.py ← subprocess wrapper, tree renderer + │ └── theme_generator.py ← deterministic CSS from namespace name + ├── api/ + │ ├── export.py ← /export/ endpoints (namespace, thread, node) + │ ├── wiki.py ← /wiki-edit, /revisions endpoints + │ ├── themes.py ← /themes/ endpoints (css, preview) + │ ├── views.py ← modified: source_format on create/reply/edit + │ └── serializers.py ← modified: source_format in node JSON + ├── models/ + │ ├── node.py ← modified: source_format column, set_data(), wiki_edit() + │ ├── namespace.py ← modified: can_wiki_edit() + │ └── revision.py ← new: Revision model + └── scripts/alembic/versions/ + ├── d47dc908d2ea_... ← add source_format to rb_node + └── 8e3c406e4049_... ← create rb_revision table +``` diff --git a/docs/testing.md b/docs/testing.md index 4a1d6cf..f33dd5a 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -100,6 +100,111 @@ curl -s -X PATCH "$REMARKBOX/api/v1/nodes/$NODE_ID" \ -d '{"data": "Updated content."}' | python3 -m json.tool ``` +## Multi-Syntax Input + +### Create Thread with RST + +```bash +curl -s -X POST "$REMARKBOX/api/v1/threads" \ + -H "Content-Type: application/json" \ + -b cookies.txt \ + -d '{ + "namespace": "meta.remarkbox.com", + "title": "RST thread", + "data": "Title\n=====\n\nA paragraph in **reStructuredText**.", + "source_format": "rst" + }' | python3 -m json.tool +``` + +### Reply with HTML + +```bash +curl -s -X POST "$REMARKBOX/api/v1/threads/$THREAD_ID/replies" \ + -H "Content-Type: application/json" \ + -b cookies.txt \ + -d '{ + "data": "

A reply in HTML.

", + "source_format": "html" + }' | python3 -m json.tool +``` + +## Export + +### List Available Formats + +```bash +curl -s "$REMARKBOX/api/v1/export/formats" | python3 -m json.tool +``` + +### Export Thread as Markdown + +```bash +curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.md" +``` + +### Export Thread as PDF + +```bash +curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.pdf" -o thread.pdf +``` + +### Export Thread as EPUB + +```bash +curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.epub" -o thread.epub +``` + +### Export Namespace as Book + +```bash +curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.epub" -o meta.epub +curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.pdf" -o meta.pdf +curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.md" +``` + +### Export Node Subtree + +```bash +curl -s "$REMARKBOX/api/v1/export/nodes/$NODE_ID.html" +``` + +## Wiki Mode + +### Wiki Edit a Node + +```bash +curl -s -X POST "$REMARKBOX/api/v1/nodes/$NODE_ID/wiki-edit" \ + -H "Content-Type: application/json" \ + -b cookies.txt \ + -d '{"data": "Updated wiki content."}' | python3 -m json.tool +``` + +### Get Revision History + +```bash +curl -s "$REMARKBOX/api/v1/nodes/$NODE_ID/revisions" | python3 -m json.tool +``` + +### Get Specific Revision + +```bash +curl -s "$REMARKBOX/api/v1/revisions/$REVISION_ID" | python3 -m json.tool +``` + +## Themes + +### Get Namespace Theme CSS + +```bash +curl -s "$REMARKBOX/api/v1/themes/meta.remarkbox.com/css" +``` + +### Preview Theme Palette + +```bash +curl -s "$REMARKBOX/api/v1/themes/meta.remarkbox.com/preview" | python3 -m json.tool +``` + ## Error Cases ### Missing namespace diff --git a/docs/tickets/15.md b/docs/tickets/15.md index 7985f8b..0599ff4 100644 --- a/docs/tickets/15.md +++ b/docs/tickets/15.md @@ -1,8 +1,9 @@ # T15: Operation Undigg — Pandoc Export & Wiki Mode -**Status**: in-progress +**Status**: resolved **Priority**: high **Source**: fox directive 2026-03-09 +**Resolved**: 2026-03-10 (`6d1cfff`) ## Summary @@ -87,34 +88,47 @@ With JS: dropdown/popover with format picker, async download. ## Phases ### Phase 1: Export pipeline (pandoc integration) -- [ ] Tree-to-markdown renderer (walk node tree → single markdown document) -- [ ] Export API endpoints (namespace, thread, node) -- [ ] Pandoc subprocess wrapper -- [ ] Format negotiation (URI suffix or Accept header) +- [x] Tree-to-markdown renderer (walk node tree → single markdown document) +- [x] Export API endpoints (namespace, thread, node) +- [x] Pandoc subprocess wrapper +- [x] Format negotiation (URI suffix) ### Phase 2: Multi-syntax input -- [ ] Add `source_format` column to Node -- [ ] Alembic migration -- [ ] Modify `set_data()` to use pandoc for non-markdown formats -- [ ] Accept `source_format` on create/reply/edit API endpoints -- [ ] HTML stripping for HTML input +- [x] Add `source_format` column to Node +- [x] Alembic migration (`d47dc908d2ea`) +- [x] Modify `set_data()` to use pandoc for non-markdown formats +- [x] Accept `source_format` on create/reply/edit API endpoints +- [x] HTML stripping for HTML input (pandoc html→markdown round-trip) ### Phase 3: Wiki mode + revisions -- [ ] Create `rb_revision` model + migration -- [ ] Implement wiki edit permissions in Namespace -- [ ] Store revisions on edit -- [ ] Revision history API endpoint -- [ ] Diff endpoint +- [x] Create `rb_revision` model + migration (`8e3c406e4049`) +- [x] Implement wiki edit permissions in Namespace (`can_wiki_edit()`) +- [x] Store revisions on edit (`node.wiki_edit()`) +- [x] Revision history API endpoint +- [ ] Diff endpoint (deferred — future work) ### Phase 4: Auto-generated themes -- [ ] Per-namespace theme generation (light + dark) -- [ ] CSS custom properties for theming -- [ ] Theme preview +- [x] Per-namespace theme generation (light + dark) +- [x] CSS custom properties for theming (`--rb-*`) +- [x] Theme preview (JSON palette) + +## Test Coverage + +95 new tests across 4 test files (544 total): + +| File | Tests | Coverage | +|------|-------|----------| +| `test_pandoc.py` | 33 | pandoc convert, formats, tree render, namespace render | +| `test_theme_generator.py` | 15 | hue, seed, CSS generation, light/dark mode | +| `test_revision.py` | 11 | Revision model, wiki edit permissions | +| `test_undigg.py` | 34 | integration tests for export, wiki, themes, multi-syntax | + +Module coverage: 76% on new code (100% on themes, serializers; 80% pandoc; 72% export; 55% wiki). ## Notes - Pandoc 3.1.3 installed at `/usr/bin/pandoc` - 43 input formats, 67 output formats -- PDF requires LaTeX (`pdflatex`) or `wkhtmltopdf` — check availability -- `Namespace.wiki` column already exists in schema, just needs implementation -- Export cache invalidation: on any node edit in the tree +- PDF via `wkhtmltopdf` (no pdflatex) +- `Namespace.wiki` column already existed in schema +- Diff endpoint deferred — revision data is stored, diffing can be added later diff --git a/docs/tickets/index.md b/docs/tickets/index.md index 2224d3f..14a3553 100644 --- a/docs/tickets/index.md +++ b/docs/tickets/index.md @@ -19,4 +19,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02- | [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` | | [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` | | [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 | -| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | in-progress | high | fox directive 2026-03-09 | +| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | resolved | high | fox directive 2026-03-09 |