twine 6 (Sep 2025) auto-detects GitLab CI and refuses to fall back to
~/.pypirc, requiring PYPI_ID_TOKEN (Trusted Publishing OIDC). Pin <6
to keep the runner's ~/.pypirc fallback working until we migrate all
python/* repos to Trusted Publishing as a coordinated change.
twine check validates the built artifacts' metadata (long_description
rendering, required fields) so a broken render hits CI logs instead of
landing on PyPI as a malformed project page.
--non-interactive guards against a wedged prompt hanging the pipeline.
The create-form selector forced authors to remember to switch from
markdown to rst before typing. Auto-detect runs on every preview
keystroke + at form submit time, picking the right format from
content shape so authors just write.
SOURCE_FORMAT_DETECTORS in custom.js carries the registry — first
match wins, ordered most-specific first. Today: html, latex,
mediawiki, rst, with markdown as the catch-all default.
Adding a new format (e.g. asciidoc, org, ipynb):
1. Push one entry to SOURCE_FORMAT_DETECTORS in custom.js
2. Add one <option> to source_format_select in create.j2
3. Confirm pandoc accepts the format name
No Python changes needed for new formats — the existing
preview_post + new_thread paths pass source_format through to
pandoc untouched.
UX:
- Default selector value: 'auto' (was 'markdown')
- "detected: rst" label appears next to selector when auto picks
- Manual override options stay for ambiguous content
- Form onsubmit resolves 'auto' to a concrete format before POST
so node.source_format never persists as the literal 'auto'
Defense in depth: new_thread.py + preview_post both treat 'auto'
as a no-op (falls to markdown default) for no-JS clients or racy
submits.
The preview fix (64a15b6) honored source_format only when the textarea
already declared one — which works for editing an existing node but
not for creating a new one. Authors writing RST in the create form
saw their preview rendered as markdown, then saved into a node whose
source_format defaulted to markdown — so even the final saved page
came back wrong.
Three pieces:
- create.j2: <select> for source_format alongside the textarea.
Default markdown; rst, html, mediawiki, latex, org, asciidoc,
textile available. onchange syncs the chosen value into
textarea.dataset.sourceFormat and re-runs preview, so the
existing custom.js sendPreview picks it up unchanged.
- new_thread.py: read source_format from request.params, pass to
set_data() so the created node's persistent source_format matches
what the author saw in preview.
- No JS change needed — custom.js already reads
textarea.dataset.sourceFormat.
Root cause of recurring 35929 / 38536 CI flake — `-n auto` on a 32-core
CI runner spawns 32 pytest workers. test_pandoc.py contains ~14 tests
that each fork a `pandoc` subprocess. When several land on parallel
workers at once, pandoc's cold-start cost (GHC runtime + filter loading)
plus runner CPU contention pushes wall time past the 5s metadata timeout
and 30s convert timeout. Subprocess gets SIGKILL'd (returncode -9),
tests fail, pipeline reruns the same flake.
Three-part root cause fix:
1. Cache get_available_input_formats / get_available_output_formats with
functools.lru_cache(maxsize=1). Pandoc's format list is static for
a given binary; we only need one subprocess per Python process,
not one per call site. Also bump the metadata-query timeout from 5s
to 30s — the first cold-start under contention still has to succeed.
2. Mark TestConvert + TestAvailableFormats with
@pytest.mark.xdist_group("pandoc-subprocess") so every pandoc-forking
test pins to one xdist worker. Other 31 workers continue parallel-
processing the rest of the suite; pandoc tests run sequentially on
their single worker, each pandoc cold-start completes before the
next starts.
3. Update Makefile test target to pass --dist=loadgroup so pytest-xdist
honors the xdist_group marker.
Other test classes in test_pandoc.py (TestNodeTreeToMarkdown,
TestNamespaceToMarkdown, TestGetAuthorName) use MockNode objects
without data_html, so node_tree_to_markdown's pandoc path stays cold —
those classes don't need the marker.
Replaces 1527ef7 (which just retriggered the same flake).
The /preview-post endpoint hardcoded markdown_to_html(), ignoring the
node's source_format. RST pages showed `..` comments verbatim, literal
`name_` references, and unrendered `.. _target: url` definitions, even
though Node.set_data dispatched correctly on save. Preview lied; save
told the truth.
Mirror set_data's dispatch in preview_post:
- markdown → markdown_to_html (unchanged)
- html → pandoc clean → markdown_to_html (matches set_data)
- any other → pandoc + namespace sanitizer
Plumb source_format from the edit textarea via data-source-format,
read by sendPreview in custom.js, sent as a form param.
setuptools 81 removed pkg_resources from its distribution. pyramid 2.0.x
(and 2.1) still does `import pkg_resources`. When setuptools 82.0.1 landed
in /opt/remarkbox/env, pyramid failed to import, all three uwsgi services
on origin crash-looped, and my.remarkbox.com / meta.remarkbox.com /
foxhop.net / westworld2.com served 502.
remarkbox/_vendor/pkg_resources/__init__.py is a 115-line shim backed by
importlib.resources (stdlib only — no setuptools, no jaraco.text, no
platformdirs). Exposes exactly the surface pyramid uses:
resource_filename / resource_stream / resource_string / resource_exists /
resource_isdir / resource_listdir / DefaultProvider / register_loader_type.
remarkbox/__init__.py prepends our _vendor dir to sys.path before the
first pyramid import so `import pkg_resources` always finds our shim,
regardless of which setuptools is installed. Bleeding-edge friendly:
upstream setuptools removals can no longer break us.
558 tests pass.
68ca0ef swapped common.css :root from the Remarkbox light palette to the
chaostheory dark palette (teal background, ice-blue primary) and dark-skinned
.alert-*, section.well, .focused, etc. common.css ships in every theme, so
chaostheory colors leaked into meta/default pages (e.g. the teal well box on
my.remarkbox.com). Reverted those rules to their pre-68ca0ef values; the
chaostheory palette now lives in remarkbox-theme-chaostheory/theme.css. Kept
the .export-menu styles added in 9e22c0d.
Edge proxy rewrites Host: www.foxhop.net → foxhop.net before forwarding
to origin (see proxy.unturf.com Caddyfile). request.host on origin is
therefore always the apex, stripping any www. prefix even when the user
fetched from www. Caddy's reverse_proxy preserves the original host in
X-Forwarded-Host, so consult that first and fall back to request.host
when the proxy isn't in the path.
Provenance helpers were using namespace.name as the host. Foxhop's
namespace is named 'foxhop.net' but the public URI lives at
'www.foxhop.net' — exports were emitting 'https://foxhop.net/...' which
works (apex 301-redirects to www) but reads wrong on a printed page.
Pass request.host through into the provenance bundle so canonical URIs,
chapter titles, and per-reply permalinks all preserve whatever host
prefix the export was actually fetched from. Falls back to namespace.name
when no host is supplied (callers outside the request context).
Pandoc's --standalone HTML5 template applies max-width: 36em + margin: 0
auto + padding: 50px to body, centering content in a narrow column on
any page size. With wkhtmltopdf at 15mm page margins, effective content
margins were measuring ~55mm because pandoc's CSS added ~40mm of inner
padding on top.
Inject a header-includes <style> block that strips body's max-width,
margin, and padding so content fills the page minus wkhtmltopdf's own
page margins. Measured: 55mm → 15mm left margin.
wkhtmltopdf default top margin is ~25mm — leaves a half-page of empty
space above the provenance header on every PDF. Set explicit 12mm
top/bottom and 15mm left/right via pandoc -V margin-* options.
Shorten QR caption from 'Scan to visit the living source' (six words,
wrapped onto four lines in the 160px cell) to 'Scan for living source'
(four words). Widen the QR cell from 160px to 200px so the caption
sits on one or two lines without crowding the QR.
Local pytest run shows 584 pass on the same code; previous CI test
job hit a transient failure. Retry to validate the auto-restart fix in
foxhop-states 591cb4f end-to-end through CI.
Empty commit triggers a salt highstate so the new auto-restart cmd.run
in uwsgi.caddy_sites lands and (assuming akuma's clone of foxhop-states
has been pulled to 591cb4f) fires the catch-up restart of foxhop.net
and westworld2.com onto whatever commit hash this build produces.
Wraps the provenance header in an HTML table so the source/snapshot/
generator block sits left and the QR sits right at the same horizontal
level. Pandoc converts the table cleanly into native table cells across
HTML, PDF (wkhtmltopdf), DOCX, ODT, EPUB.
Without a QR (qr_data_uri=None) we fall back to the plain blockquote.
Triggers a salt highstate so the content-based skip_deploy auto-restart
state lands on origin and finally drives the catch-up restart of
westworld2 (:6002) and foxhop.net (:6003) onto the current commit.
Every exported document (PDF, EPUB, DOCX, HTML, markdown, plain, ...)
now points back to its living source on Remarkbox so a printed snapshot
remains traceable years later.
What's injected:
- Top banner: source URI, snapshot ISO timestamp, generator commit hash,
plus a notice the document is a snapshot of a living source.
- QR code (PNG, embedded as data URI) linking back to the canonical URI.
Scannable from print, survives the pandoc image pipeline across
HTML, PDF (via wkhtmltopdf), DOCX, EPUB, ODT.
- Per-reply permalinks: each reply heading hyperlinks its date to the
deep-link permalink (thread-uri#node-id), academic-citation style.
- Footer repeating the source URI + commit hash.
- Namespace exports also hyperlink each chapter title to the live
thread.
Canonical URI = https://{namespace.name}{node.path} — namespace name is
its host on the public web.
New module: remarkbox/lib/provenance.py — header_md, footer_md,
reply_heading_md, qr_png_data_uri, canonical_uri helpers, and a one-shot
build() bundler.
Dep added: segno (pure-Python QR generator, BSD, zero deps).
Export menu: dropped the details/summary toggle so format options are
visible at all times; added flex layout with gap for breathing room.
Wiki history: the browser edit form path was calling node.edit() instead
of node.wiki_edit() so revisions were never recorded — /api/v1/nodes/.../revisions
returned empty after every edit. Also tightened the access gate to
can_wiki_edit so non-owner authenticated users can edit wiki-mode root
nodes through the UI (matching the gate already used by the edit-button
macro).
Observed defect: a node's source_format column can disagree with the
actual bytes in data. A Collatz thread on foxhop.net has data filled
with RST (======, ---, :: blocks) yet source_format="markdown". The
earlier helper trusted source_format and returned the RST untouched,
so GET /api/v1/export/.../thread.md served raw RST to anyone who asked
for markdown.
data_html is the canonical rendered form — set at every write via
set_data() regardless of input syntax — so converting data_html → markdown
via pandoc sidesteps the label mismatch entirely. Falls back to raw
data if data_html is missing or pandoc fails.
Three fixes to thread/namespace export:
1. Nodes authored in non-markdown source_format (rst, mediawiki, latex, ...)
had their raw source dumped directly into the markdown renderer. A RST
thread exported as .md therefore yielded RST, not markdown. New
_node_data_as_markdown helper converts node.data through pandoc when
source_format != markdown, with a raw-source fallback on pandoc failure.
2. node_tree_to_markdown prepended '# {root_node.title}' on top of data
that already carries its own H1 (either native markdown or a converted
RST underline heading). Drop the prepend — the data owns the title.
3. convert() passed --metadata title=X which makes pandoc render a visible
title-block in HTML/PDF above the body. Combined with (2) and the data's
own H1 this showed the title three times in exported HTML/PDF. For
html-family outputs (and pdf via wkhtmltopdf) switch to -V pagetitle=X
so only the <title> tag gets populated; other formats still use the
proper --metadata title=X for real document metadata.
- New route /{node_id}/revisions rendered by revision-history.j2
- Show content/history tabs on wiki root nodes
- Export dropdown includes history (json + html) links on wiki namespaces
- Node action buttons now gate on can_wiki_edit (not can_alter_node)
so wiki-mode members can edit their own root nodes
- Suppress Topic link when URI matches current domain (self-reference)
- Tests covering revision timestamps, wiki_edit(), can_wiki_edit(),
/revisions endpoints, CSRF trusted origins, export menu gating,
pandoc rst→html rendering
moderators property returned self.enabled_users (every enabled member),
so every user who had ever posted got moderator powers: spam
approve/deny, edit/delete other users' comments, moderator Slack
notifications. Bug was universal across all namespaces — any site where
users post was affected.
Fix: moderators now returns only users with role='moderator'.
Owners now implicitly pass is_moderator() checks, since NamespaceUser
is single-role per user per namespace; this keeps owners functional
without needing a data migration.
Adds chaostheory theme to the shared env.tar.gz build so foxhop.net
(which sets app.theme = chaostheory and shares /opt/remarkbox/env with
my.remarkbox.com) has the theme's templates + static assets available
after deploy.
- views/__init__.py: add MAX_CONTENT_LENGTH = 500_000 (shared constant with comment
explaining the CWE-407 / bleach ReDoS rationale)
- reply_node.py: reject oversized content before set_data() / clean_raw_html()
- modify_node.py: same guard on edit path
- new_thread.py: same guard on new thread path
- requirements.py3.txt: tighten bleach>=2.1.4 -> bleach>=6.0.0 with CVE note
- test_render.py: unit tests for bleach version contract, API stability,
sanitization correctness, and ReDoS resistance timing
- test_views.py: functional tests for content length enforcement on all three
browser form paths (reply, new thread)
- list_nodes.py: strip empty tokens, cap keywords to 10 before passing to model
- node.py: add .limit(200) per keyword query — was unbounded .all()
- namespace.py: add .limit(500) on root.children in dict_dump — was unbounded iteration
- __init__.py: cap page_number to 1000 — large offsets force full table scans
Diff: GET /api/v1/revisions/{id}/diff/{other_id} — unified diff between two revisions.
UI: <details>-based export menus on thread and node views (works without JS).
UI: Revision history link for wiki-mode namespaces.
Maps which domains go through the edge proxy (142.93.73.64) vs direct to
origin (162.243.167.224). Prevents repeat of the 5-day meta/faq SSL outage
caused by missing proxy blocks.
