Update docs for Operation Undigg: API reference, testing guide, architecture diagram.
This commit is contained in:
parent
6d1cfff91d
commit
b63277e4d1
5 changed files with 488 additions and 23 deletions
214
docs/api.md
214
docs/api.md
|
|
@ -190,6 +190,7 @@ Request body:
|
|||
- `namespace` (required)
|
||||
- `title` (required)
|
||||
- `data` (required, max 500000 chars)
|
||||
- `source_format` (optional, default `"markdown"`) — any pandoc input format
|
||||
- `anonymous_name` (optional, used when namespace allows anonymous)
|
||||
- `email` (optional, creates an unverified user)
|
||||
|
||||
|
|
@ -220,6 +221,7 @@ Request body:
|
|||
```
|
||||
|
||||
- `data` (required, max 500000 chars)
|
||||
- `source_format` (optional, default `"markdown"`)
|
||||
- `anonymous_name` (optional)
|
||||
- `email` (optional)
|
||||
|
||||
|
|
@ -264,12 +266,14 @@ Request body:
|
|||
```json
|
||||
{
|
||||
"data": "Updated markdown",
|
||||
"title": "Updated Title"
|
||||
"title": "Updated Title",
|
||||
"source_format": "markdown"
|
||||
}
|
||||
```
|
||||
|
||||
- `data` (optional, updates content)
|
||||
- `title` (optional, only applies to root nodes)
|
||||
- `source_format` (optional, default `"markdown"`)
|
||||
|
||||
At least one of `data` or `title` is required.
|
||||
|
||||
|
|
@ -350,6 +354,214 @@ Error `401`:
|
|||
}
|
||||
```
|
||||
|
||||
### Multi-Syntax Input
|
||||
|
||||
All write endpoints (`POST /threads`, `POST /replies`, `PATCH /nodes`) accept
|
||||
an optional `source_format` parameter. Default is `"markdown"`.
|
||||
|
||||
Supported input formats include any pandoc-supported format: `markdown`, `html`,
|
||||
`rst`, `mediawiki`, `latex`, `textile`, `org`, `docbook`, `commonmark`, etc.
|
||||
|
||||
```json
|
||||
{
|
||||
"data": "Title\n=====\n\nA paragraph in reStructuredText.",
|
||||
"source_format": "rst"
|
||||
}
|
||||
```
|
||||
|
||||
HTML input is round-tripped through pandoc (html → markdown) to produce a clean
|
||||
canonical source. All formats are rendered to HTML via pandoc and sanitized
|
||||
through the bleach pipeline before storage.
|
||||
|
||||
The `source_format` field is included in all node serializations.
|
||||
|
||||
---
|
||||
|
||||
### Export Formats
|
||||
|
||||
```
|
||||
GET /api/v1/export/formats
|
||||
```
|
||||
|
||||
Returns all available pandoc output formats.
|
||||
|
||||
Response `200`:
|
||||
```json
|
||||
{
|
||||
"formats": ["asciidoc", "commonmark", "docx", "epub", "html5", "latex", "markdown", "pdf", "rst", "..."],
|
||||
"count": 67
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Export Thread
|
||||
|
||||
```
|
||||
GET /api/v1/export/threads/{node_id}.{format}
|
||||
```
|
||||
|
||||
Exports a single thread (root + replies) as a document.
|
||||
|
||||
Examples:
|
||||
```
|
||||
GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.pdf
|
||||
GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.epub
|
||||
GET /api/v1/export/threads/9f970183-ffaf-11f0-b565-040140774501.md
|
||||
```
|
||||
|
||||
Binary formats (pdf, epub, docx) return the file with `Content-Disposition: attachment`.
|
||||
Text formats return inline with appropriate content type.
|
||||
|
||||
---
|
||||
|
||||
### Export Namespace
|
||||
|
||||
```
|
||||
GET /api/v1/export/namespace/{namespace_name}.{format}
|
||||
```
|
||||
|
||||
Exports an entire namespace as a book. Each root thread becomes a chapter.
|
||||
|
||||
Examples:
|
||||
```
|
||||
GET /api/v1/export/namespace/meta.remarkbox.com.epub
|
||||
GET /api/v1/export/namespace/meta.remarkbox.com.pdf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Export Node (On-Demand)
|
||||
|
||||
```
|
||||
GET /api/v1/export/nodes/{node_id}.{format}
|
||||
```
|
||||
|
||||
Exports any node and its subtree. Useful for exporting a specific subthread
|
||||
at any nesting depth.
|
||||
|
||||
---
|
||||
|
||||
### Wiki Edit
|
||||
|
||||
```
|
||||
POST /api/v1/nodes/{node_id}/wiki-edit
|
||||
```
|
||||
|
||||
Wiki-edit a root node. Creates a revision snapshot before applying the edit.
|
||||
Requires authentication. The namespace must have `wiki = True`, or the user
|
||||
must be the node owner/moderator.
|
||||
|
||||
Request body:
|
||||
```json
|
||||
{
|
||||
"data": "Updated wiki content",
|
||||
"source_format": "markdown"
|
||||
}
|
||||
```
|
||||
|
||||
Response `200`:
|
||||
```json
|
||||
{
|
||||
"node": {"id": "...", "data": "Updated wiki content", "...": "..."},
|
||||
"revision": {"id": "...", "revision_number": 2, "...": "..."}
|
||||
}
|
||||
```
|
||||
|
||||
Errors:
|
||||
- `401` if not authenticated
|
||||
- `403` if wiki editing not allowed for this user/node
|
||||
- `404` if node not found
|
||||
|
||||
---
|
||||
|
||||
### Node Revisions
|
||||
|
||||
```
|
||||
GET /api/v1/nodes/{node_id}/revisions
|
||||
```
|
||||
|
||||
Returns the revision history for a node.
|
||||
|
||||
Response `200`:
|
||||
```json
|
||||
{
|
||||
"node_id": "...",
|
||||
"revisions": [
|
||||
{
|
||||
"id": "...",
|
||||
"revision_number": 1,
|
||||
"data": "Original content",
|
||||
"source_format": "markdown",
|
||||
"created": 1710043200000,
|
||||
"user": {"id": "...", "name": "timehexon"}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Get Revision
|
||||
|
||||
```
|
||||
GET /api/v1/revisions/{revision_id}
|
||||
```
|
||||
|
||||
Returns a specific revision by ID.
|
||||
|
||||
Response `200`:
|
||||
```json
|
||||
{
|
||||
"revision": {
|
||||
"id": "...",
|
||||
"node_id": "...",
|
||||
"revision_number": 1,
|
||||
"data": "Content at this revision",
|
||||
"source_format": "markdown",
|
||||
"created": 1710043200000,
|
||||
"user": {"id": "...", "name": "timehexon"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Theme CSS
|
||||
|
||||
```
|
||||
GET /api/v1/themes/{namespace_name}/css
|
||||
```
|
||||
|
||||
Returns auto-generated CSS theme for a namespace. Deterministic — same
|
||||
namespace always produces the same theme. Includes light mode (`:root`,
|
||||
`.theme-light`) and dark mode (`@media (prefers-color-scheme: dark)`,
|
||||
`.theme-dark`).
|
||||
|
||||
Response: `200 text/css` with 1-day cache header.
|
||||
|
||||
---
|
||||
|
||||
### Theme Preview
|
||||
|
||||
```
|
||||
GET /api/v1/themes/{namespace_name}/preview
|
||||
```
|
||||
|
||||
Returns the theme color palette as JSON for previewing without loading CSS.
|
||||
|
||||
Response `200`:
|
||||
```json
|
||||
{
|
||||
"namespace": "meta.remarkbox.com",
|
||||
"hue": 217,
|
||||
"light": {"bg": "#f8f9fa", "text": "#1a1a2e", "link": "#2563eb", "...": "..."},
|
||||
"dark": {"bg": "#0f0f1a", "text": "#e8e8f0", "link": "#60a5fa", "...": "..."}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error Format
|
||||
|
||||
All errors return a JSON body with an `error` key:
|
||||
|
|
|
|||
134
docs/architecture-undigg.md
Normal file
134
docs/architecture-undigg.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# Operation Undigg — Architecture
|
||||
|
||||
Document-first platform: every namespace is a book, every thread is a chapter,
|
||||
every reply is a section. Pandoc renders 67 output formats.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Write Path │
|
||||
│ │
|
||||
User Input │ source_format pandoc bleach │
|
||||
(md/html/rst/ │ ─────────────► convert to ──► sanitize │
|
||||
mediawiki/latex/ │ html5 pipeline │
|
||||
textile/org/...) │ │ │ │
|
||||
│ │ ▼ │
|
||||
│ │ data_html │
|
||||
│ │ (rendered) │
|
||||
│ │ │
|
||||
│ html input: │ │
|
||||
│ pandoc html→md │ │
|
||||
│ (round-trip clean) │ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ data (raw) │ │
|
||||
│ + source_format │ │
|
||||
└─────────────────────────────────────────────┘
|
||||
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Read / Export Path │
|
||||
│ │
|
||||
Node tree │ tree-to-markdown pandoc │
|
||||
(adjacency list) ────► │ renderer ──► convert to ────► output
|
||||
│ target format │
|
||||
│ │
|
||||
│ Hierarchy: │
|
||||
│ # Thread Title (h1) │
|
||||
│ ## Author Name (h2, per reply) │
|
||||
│ ### Author Name (h3, nested reply) │
|
||||
│ ...depth maps to heading level │
|
||||
└─────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Export Hierarchy
|
||||
|
||||
```
|
||||
Namespace (book)
|
||||
GET /api/v1/export/namespace/{name}.{fmt}
|
||||
│
|
||||
├── Root Node A (chapter) ← default export
|
||||
│ GET /api/v1/export/threads/{id}.{fmt}
|
||||
│ │
|
||||
│ ├── Reply 1 (section) ← on-demand
|
||||
│ │ GET /api/v1/export/nodes/{id}.{fmt}
|
||||
│ │ │
|
||||
│ │ └── Reply 1.1 (subsection) ← on-demand
|
||||
│ │
|
||||
│ └── Reply 2 (section) ← on-demand
|
||||
│
|
||||
└── Root Node B (chapter) ← default export
|
||||
GET /api/v1/export/threads/{id}.{fmt}
|
||||
```
|
||||
|
||||
**67 output formats** including: markdown, html5, pdf, epub, docx, odt, rst,
|
||||
latex, mediawiki, man, plain, rtf, asciidoc, textile, org, json, and more.
|
||||
|
||||
## Wiki Mode
|
||||
|
||||
```
|
||||
┌──────────────┐ wiki_edit() ┌──────────────┐
|
||||
│ │ ──────────────────► │ rb_revision │
|
||||
│ rb_node │ │ │
|
||||
│ │ 1. snapshot current │ id │
|
||||
│ id │ data → revision │ node_id (FK) │
|
||||
│ data │ 2. increment rev# │ user_id (FK) │
|
||||
│ data_html │ 3. overwrite node │ data │
|
||||
│ source_fmt │ with new content │ source_format │
|
||||
│ │ │ revision_number│
|
||||
└──────────────┘ │ created │
|
||||
│ └──────────────┘
|
||||
│
|
||||
▼
|
||||
can_wiki_edit(node, user)?
|
||||
├── user.authenticated? → required
|
||||
├── namespace.can_alter_node()? → owner/moderator always yes
|
||||
└── namespace.wiki && node.is_root? → wiki mode for root nodes
|
||||
```
|
||||
|
||||
## Auto-Generated Themes
|
||||
|
||||
```
|
||||
namespace_name
|
||||
│
|
||||
▼
|
||||
SHA-256 hash
|
||||
│
|
||||
├── bits[0:8] → hue (0-360)
|
||||
├── bits[8:40] → 8 seed values
|
||||
│
|
||||
▼
|
||||
HSL color palette
|
||||
│
|
||||
├── Light mode (:root, .theme-light)
|
||||
│ --rb-bg, --rb-text, --rb-link, --rb-accent, --rb-border, ...
|
||||
│
|
||||
└── Dark mode (@media prefers-color-scheme: dark, .theme-dark)
|
||||
--rb-bg, --rb-text, --rb-link, --rb-accent, --rb-border, ...
|
||||
|
||||
Deterministic: same namespace → same theme, always.
|
||||
Served at: GET /api/v1/themes/{namespace}/css (1-day cache)
|
||||
Preview: GET /api/v1/themes/{namespace}/preview (JSON palette)
|
||||
```
|
||||
|
||||
## Module Map
|
||||
|
||||
```
|
||||
remarkbox/
|
||||
├── lib/
|
||||
│ ├── pandoc.py ← subprocess wrapper, tree renderer
|
||||
│ └── theme_generator.py ← deterministic CSS from namespace name
|
||||
├── api/
|
||||
│ ├── export.py ← /export/ endpoints (namespace, thread, node)
|
||||
│ ├── wiki.py ← /wiki-edit, /revisions endpoints
|
||||
│ ├── themes.py ← /themes/ endpoints (css, preview)
|
||||
│ ├── views.py ← modified: source_format on create/reply/edit
|
||||
│ └── serializers.py ← modified: source_format in node JSON
|
||||
├── models/
|
||||
│ ├── node.py ← modified: source_format column, set_data(), wiki_edit()
|
||||
│ ├── namespace.py ← modified: can_wiki_edit()
|
||||
│ └── revision.py ← new: Revision model
|
||||
└── scripts/alembic/versions/
|
||||
├── d47dc908d2ea_... ← add source_format to rb_node
|
||||
└── 8e3c406e4049_... ← create rb_revision table
|
||||
```
|
||||
105
docs/testing.md
105
docs/testing.md
|
|
@ -100,6 +100,111 @@ curl -s -X PATCH "$REMARKBOX/api/v1/nodes/$NODE_ID" \
|
|||
-d '{"data": "Updated content."}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Multi-Syntax Input
|
||||
|
||||
### Create Thread with RST
|
||||
|
||||
```bash
|
||||
curl -s -X POST "$REMARKBOX/api/v1/threads" \
|
||||
-H "Content-Type: application/json" \
|
||||
-b cookies.txt \
|
||||
-d '{
|
||||
"namespace": "meta.remarkbox.com",
|
||||
"title": "RST thread",
|
||||
"data": "Title\n=====\n\nA paragraph in **reStructuredText**.",
|
||||
"source_format": "rst"
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Reply with HTML
|
||||
|
||||
```bash
|
||||
curl -s -X POST "$REMARKBOX/api/v1/threads/$THREAD_ID/replies" \
|
||||
-H "Content-Type: application/json" \
|
||||
-b cookies.txt \
|
||||
-d '{
|
||||
"data": "<p>A reply in <strong>HTML</strong>.</p>",
|
||||
"source_format": "html"
|
||||
}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Export
|
||||
|
||||
### List Available Formats
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/formats" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Export Thread as Markdown
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.md"
|
||||
```
|
||||
|
||||
### Export Thread as PDF
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.pdf" -o thread.pdf
|
||||
```
|
||||
|
||||
### Export Thread as EPUB
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/threads/$THREAD_ID.epub" -o thread.epub
|
||||
```
|
||||
|
||||
### Export Namespace as Book
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.epub" -o meta.epub
|
||||
curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.pdf" -o meta.pdf
|
||||
curl -s "$REMARKBOX/api/v1/export/namespace/meta.remarkbox.com.md"
|
||||
```
|
||||
|
||||
### Export Node Subtree
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/export/nodes/$NODE_ID.html"
|
||||
```
|
||||
|
||||
## Wiki Mode
|
||||
|
||||
### Wiki Edit a Node
|
||||
|
||||
```bash
|
||||
curl -s -X POST "$REMARKBOX/api/v1/nodes/$NODE_ID/wiki-edit" \
|
||||
-H "Content-Type: application/json" \
|
||||
-b cookies.txt \
|
||||
-d '{"data": "Updated wiki content."}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Get Revision History
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/nodes/$NODE_ID/revisions" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Get Specific Revision
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/revisions/$REVISION_ID" | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Themes
|
||||
|
||||
### Get Namespace Theme CSS
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/themes/meta.remarkbox.com/css"
|
||||
```
|
||||
|
||||
### Preview Theme Palette
|
||||
|
||||
```bash
|
||||
curl -s "$REMARKBOX/api/v1/themes/meta.remarkbox.com/preview" | python3 -m json.tool
|
||||
```
|
||||
|
||||
## Error Cases
|
||||
|
||||
### Missing namespace
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
# T15: Operation Undigg — Pandoc Export & Wiki Mode
|
||||
|
||||
**Status**: in-progress
|
||||
**Status**: resolved
|
||||
**Priority**: high
|
||||
**Source**: fox directive 2026-03-09
|
||||
**Resolved**: 2026-03-10 (`6d1cfff`)
|
||||
|
||||
## Summary
|
||||
|
||||
|
|
@ -87,34 +88,47 @@ With JS: dropdown/popover with format picker, async download.
|
|||
## Phases
|
||||
|
||||
### Phase 1: Export pipeline (pandoc integration)
|
||||
- [ ] Tree-to-markdown renderer (walk node tree → single markdown document)
|
||||
- [ ] Export API endpoints (namespace, thread, node)
|
||||
- [ ] Pandoc subprocess wrapper
|
||||
- [ ] Format negotiation (URI suffix or Accept header)
|
||||
- [x] Tree-to-markdown renderer (walk node tree → single markdown document)
|
||||
- [x] Export API endpoints (namespace, thread, node)
|
||||
- [x] Pandoc subprocess wrapper
|
||||
- [x] Format negotiation (URI suffix)
|
||||
|
||||
### Phase 2: Multi-syntax input
|
||||
- [ ] Add `source_format` column to Node
|
||||
- [ ] Alembic migration
|
||||
- [ ] Modify `set_data()` to use pandoc for non-markdown formats
|
||||
- [ ] Accept `source_format` on create/reply/edit API endpoints
|
||||
- [ ] HTML stripping for HTML input
|
||||
- [x] Add `source_format` column to Node
|
||||
- [x] Alembic migration (`d47dc908d2ea`)
|
||||
- [x] Modify `set_data()` to use pandoc for non-markdown formats
|
||||
- [x] Accept `source_format` on create/reply/edit API endpoints
|
||||
- [x] HTML stripping for HTML input (pandoc html→markdown round-trip)
|
||||
|
||||
### Phase 3: Wiki mode + revisions
|
||||
- [ ] Create `rb_revision` model + migration
|
||||
- [ ] Implement wiki edit permissions in Namespace
|
||||
- [ ] Store revisions on edit
|
||||
- [ ] Revision history API endpoint
|
||||
- [ ] Diff endpoint
|
||||
- [x] Create `rb_revision` model + migration (`8e3c406e4049`)
|
||||
- [x] Implement wiki edit permissions in Namespace (`can_wiki_edit()`)
|
||||
- [x] Store revisions on edit (`node.wiki_edit()`)
|
||||
- [x] Revision history API endpoint
|
||||
- [ ] Diff endpoint (deferred — future work)
|
||||
|
||||
### Phase 4: Auto-generated themes
|
||||
- [ ] Per-namespace theme generation (light + dark)
|
||||
- [ ] CSS custom properties for theming
|
||||
- [ ] Theme preview
|
||||
- [x] Per-namespace theme generation (light + dark)
|
||||
- [x] CSS custom properties for theming (`--rb-*`)
|
||||
- [x] Theme preview (JSON palette)
|
||||
|
||||
## Test Coverage
|
||||
|
||||
95 new tests across 4 test files (544 total):
|
||||
|
||||
| File | Tests | Coverage |
|
||||
|------|-------|----------|
|
||||
| `test_pandoc.py` | 33 | pandoc convert, formats, tree render, namespace render |
|
||||
| `test_theme_generator.py` | 15 | hue, seed, CSS generation, light/dark mode |
|
||||
| `test_revision.py` | 11 | Revision model, wiki edit permissions |
|
||||
| `test_undigg.py` | 34 | integration tests for export, wiki, themes, multi-syntax |
|
||||
|
||||
Module coverage: 76% on new code (100% on themes, serializers; 80% pandoc; 72% export; 55% wiki).
|
||||
|
||||
## Notes
|
||||
|
||||
- Pandoc 3.1.3 installed at `/usr/bin/pandoc`
|
||||
- 43 input formats, 67 output formats
|
||||
- PDF requires LaTeX (`pdflatex`) or `wkhtmltopdf` — check availability
|
||||
- `Namespace.wiki` column already exists in schema, just needs implementation
|
||||
- Export cache invalidation: on any node edit in the tree
|
||||
- PDF via `wkhtmltopdf` (no pdflatex)
|
||||
- `Namespace.wiki` column already existed in schema
|
||||
- Diff endpoint deferred — revision data is stored, diffing can be added later
|
||||
|
|
|
|||
|
|
@ -19,4 +19,4 @@ Tracked issues from the meta.remarkbox.com and faq.remarkbox.com audit (2026-02-
|
|||
| [T12](12.md) | Reply to API-only CRUD thread confirming done | resolved | low | meta `6db01560` |
|
||||
| [T13](13.md) | Reply to lock/archive thread confirming done | resolved | low | meta `7e9d5864` |
|
||||
| [T14](14.md) | meta/faq SSL outage — missing proxy blocks | resolved | critical | postmortem 2026-02-25 |
|
||||
| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | in-progress | high | fox directive 2026-03-09 |
|
||||
| [T15](15.md) | Operation Undigg — Pandoc export & wiki mode | resolved | high | fox directive 2026-03-09 |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue