make_post_sell/CLAUDE.md

# Claude Development Notes

## Production

- Application: `https://my.makepostsell.com`
- Version check: `https://my.makepostsell.com/version`
- Prod shell: `tmux-hosts` — look for `my.makepostsell.com` (typically tmux window `0:3`) — **READ-ONLY, never deploy/fix from tmux**
- Media CDN: `plan-period-files.nyc3.cdn.digitaloceanspaces.com` (DigitalOcean Spaces)
- Deploy pipeline: `git push` → GitLab CI (test → build → deploy) → `salt-call state.highstate` on prod
- Salt states: `~/git/foxhop-states/uwsgi/` — note: MPS uses `caddy_sites.sls`, NOT `sites.sls`
- Salt pillar: `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls`
- DB path on prod: `/opt/make_post_sell/make_post_sell.sqlite` (owned by `uwsgi`, need `sudo` for writes)
- Timestamps in DB are **milliseconds** not seconds (13 digits)
- uWSGI: 2 processes, 8 threads, reload-on-rss 512MB (pillar-configurable), Caddy reverse proxy on :6001

### 🚨 NEVER Operate on Production Directly

**ABSOLUTE RULE**: ALL fixes go through CI/CD and Salt. No exceptions.

- **NEVER** SSH into prod and run `ALTER TABLE`, `sqlite3`, or any direct DB command
- **NEVER** bypass the migration system — if `alembic upgrade head` fails, fix the migration and push
- **The fix is always in the code.** Push to master → CI tests → deploy (Salt highstate or CI-direct) → Alembic runs on every instance
- Restarting services via SSH is fine for recovery, but the underlying fix must still go through code + deploy

**Why this matters**: MPS runs open source on multiple servers (makepostsell.com, memopoly.com, and any operator instance). A manual fix on one server leaves every other instance broken. The migration system exists to apply changes everywhere consistently.

**When prod has a 502**: diagnose via logs (read-only SSH is fine), fix the code, push. The deploy pipeline reaches all instances. A manual fix reaches one.

### Media Architecture

Files are NEVER streamed through uwsgi. Our server only generates presigned URLs (15 min TTL). Our client's browser/JS fetches directly from our Spaces CDN:
- **Downloads**: presigned `get_object` URLs → client fetches from CDN
- **Uploads**: presigned `post` → client uploads directly to Spaces
- **Thumbnails**: public CDN URLs with `?ts=` cache busting

**BYOB (Bring Your Own Bucket)**: Shops can configure their own S3-compatible bucket (`bucket-settings` form section). When enabled, all presigned URLs and CDN references use our shop's bucket. Always use shop-aware request methods in views and templates:
- `request.shop_uploads_client` — S3 client (shop's or MPS default)
- `request.shop_bucket_name` — bucket name (shop's or MPS default)
- `request.shop_cdn_endpoint` — CDN URL (shop's or MPS default)

**NEVER** use `request.app["bucket.secure_uploads"]`, `request.app["bucket.secure_uploads.get_endpoint"]`, or `request.secure_uploads_client` directly in views or templates. These are only used internally by `request_methods.py` as fallbacks.

### Transactional Email (lib/mail.py)

All transactional mail (OTP login codes, receipts, sale/offer notifications, gift cards, invites) sends from **one warm sending identity**: `app.email.sender` (default `no-reply@origin.makepostsell.com`, overridable via `MPS_EMAIL_SENDER`) — *not* per-shop `no-reply@<shop-domain>`. The recipient-facing name is the **shop name** when in shop context, else `app.email.from_name` (default `Make Post Sell`); `lib/mail.py:format_from_header()` builds the `From:` header. Why a single identity: operator custom-domain shops (e.g. `shop.unturf.com`) have no DKIM key MPS controls and don't authorize MPS's sending IPs in SPF, so per-domain `From:` lands in spam. `origin.makepostsell.com` is DKIM-signed by opendkim on the origin box (`d=makepostsell.com`, selector `20190727`) and SPF-authorized (`v=spf1 a a:mx1.foxhop.net -all`), and outbound is relayed through `mx1.foxhop.net` (warm IP, see `foxhop-pillar/postfix/makepostsell.sls` → `postfix_relayhost`). Reply-To / per-shop contact email is still TODO — see `docs/tickets/mps-23.md`.

### Karaoke Pipeline (lib/karaoke.py)

Disk-backed vocal isolation pipeline using spectral mid-side Wiener masking
(`voxsplit.c`, zero ML deps). Streams media to unsandbox via `POST /upload`
(64KB chunks, constant memory), executes in zerotrust container, streams
response back, uploads instrumentals + vocals to S3.

Full architecture doc: `docs/karaoke-pipeline.md` (with dot diagrams).

- **Concurrency**: `ThreadPoolExecutor` sized to account's unsandbox concurrency limit
- **Memory**: ~64KB per worker at every stage (disk-backed, not in-memory)
- **Upstream limit**: 3.698GB / 3,698,742,051 bytes per file (unsandbox `@max_upload_bytes`)
- **Retries**: 3 attempts with exponential backoff (5s, 10s)
- **Callers**: `views/product.py` (upload), `views/watch.py` (on-demand), `views/shop.py` (backfill), `scripts/backfill_karaoke.py`
- **On-demand**: `POST /karaoke/{product_id}` — forks detached child, watch.js 10s refresh detects completion, auto-switches to instrumentals
- **Streaming path**: MPS → `POST /upload` → API encrypts to disk → pool pulls via `GET /internal/upload/{id}` → pipes into container `/root/input/` — zero bytes cross Erlang distribution

## Project Setup

This project uses a Makefile for most development operations. Use `make` commands instead of running tools directly.

## Common Development Tasks

### Testing
- Run tests: `make test`
  - This installs development dependencies and runs our test suite with py.test
  - Tests are located in `make_post_sell/tests/`

### Installation & Setup
- Install from source for development: `make install-from-source`
- Install from PyPI: `make install-from-pypi`
- Initialize database: `make init-db`

### Development Server
- Start development server: `make serve`
  - Runs with auto-reload enabled
  - Uses `data/development.ini` configuration

### Environment Management
- Create virtual environment: `make venv`
- Clean up environment: `make clean`
- Activate environment: `source env/bin/activate`

## Code Structure

### Key Directories
- `make_post_sell/views/` - View controllers
- `make_post_sell/models/` - Database models
- `make_post_sell/tests/` - Test suite

### Important Files
- `make_post_sell/views/cart.py` - Cart and checkout logic
- `development.ini` - Configuration file

### Design System Files
- `static/css/tokens.css` — Design tokens (colors, typography, spacing, shape, elevation, motion, z-index), base resets, utility classes, animations. Single source of truth. Light mode `:root`, dark mode `[data-theme="dark"]`.
- `static/css/common.css` — Component styles consuming tokens via `var(--token, fallback)`.
- `templates/styleguide.j2` — Live component reference at `/styleguide` (view: `views/misc.py:23`).
- `docs/design-system.md` — Full design system reference doc (token tables, architecture diagram, conventions).

## Testing Notes

Our project uses pytest with unittest framework. There are three types of tests:

### Test Types
- **Unit tests** (`test_models.py`) - Test individual model methods and properties in isolation
- **Integration tests** (`test_integration.py`) - Test interactions between models and business logic 
- **Functional tests** (`test_functional.py`) - End-to-end tests through our web interface

### Running Tests
**Before running tests**: Source environment variables with `source vars.sh` to set required Stripe API keys and other configuration.

```bash
# Run all tests
make test

# Run specific test types
env/bin/py.test make_post_sell/tests/test_models.py     # Unit tests
env/bin/py.test make_post_sell/tests/test_integration.py # Integration tests  
env/bin/py.test make_post_sell/tests/test_functional.py  # Functional tests

# Run with coverage
env/bin/py.test --cov=make_post_sell.models.cart --cov-report=term-missing make_post_sell/tests/test_models.py::TestCart
```

### Current Coverage (712 tests)
- Cart model unit tests cover critical business logic like `requires_payment` threshold (64 cents)
- Shop environment, trial, and BYOB model properties (TestShopEnvironment, TestShopTrial, TestShopBYOB)
- Gift card model unit tests (generation, validation, transactions)
- Integration tests verify free coupon checkout, gift card flows, and multi-model interactions
- Functional tests cover cart/checkout/payment, gift card settings, environment settings, bucket settings

## Database Location

Our SQLite database is located at: `data/make_post_sell.sqlite`

**CRITICAL WARNING**: NEVER delete or remove database files without explicit user permission. Our database contains production data and cannot be easily recovered. Always ask before any destructive operations.

**MANDATORY**: ALWAYS create a backup of our database before any database operations (migrations, schema changes, etc.):
```bash
cp data/make_post_sell.sqlite data/make_post_sell.sqlite.backup-$(date +%Y%m%d-%H%M%S)
```

Query crypto payments: 
```sql
-- Note: Remove dashes from UUIDs when querying
SELECT * FROM mps_crypto_payment WHERE id = 'paymentuuidherewithoutdashes';
```

## Database Migrations

When making changes to database models, always create Alembic migrations:

### Creating Migrations

**CRITICAL**: ALWAYS use `make migration` to generate migration files. NEVER manually create migration files. NEVER hand-write or invent revision IDs. Alembic generates cryptographically unique revision IDs — a made-up ID like `a1b2c3d4e5f6` will corrupt the migration chain and break production deploys.

```bash
# The ONLY correct way to create a migration:
make migration m="description of change"
# → writes make_post_sell/scripts/alembic/versions/05be3044c2d2_description_of_change.py
# → revision ID is auto-generated (e.g. 05be3044c2d2), never invent one

# Apply pending migrations:
make migrate

# Check status:
make migration-status
```

If `make` is not available, the raw command is:
```bash
env/bin/alembic -c data/development.ini revision --autogenerate -m "description of change"
```

The generated file lives in `make_post_sell/scripts/alembic/versions/`. Edit it to add `_column_exists` / `_table_exists` guards (see idempotent pattern below), then commit it.

### Running Migrations
```bash
# Apply all pending migrations
alembic -c data/development.ini upgrade head

# Check current migration status
alembic -c data/development.ini current

# View migration history
alembic -c data/development.ini history
```

**IMPORTANT**: Always backup our database before running migrations!

### Important Migration Notes

**Idempotent Migrations**: `make init-db` creates all tables from models, so migrations that run afterward must not fail if tables/columns already exist. Always guard `create_table` with `_table_exists` and `add_column` with `_column_exists`:

```python
def _table_exists(name):
    conn = op.get_bind()
    result = conn.execute(
        sa.text("SELECT name FROM sqlite_master WHERE type='table' AND name=:name"),
        {"name": name},
    )
    return result.fetchone() is not None


def _column_exists(table, column):
    conn = op.get_bind()
    result = conn.execute(sa.text(f"PRAGMA table_info({table})"))
    return any(row[1] == column for row in result.fetchall())


def upgrade():
    if not _table_exists("mps_new_table"):
        op.create_table(...)

    if not _column_exists("mps_shop", "new_column"):
        op.add_column(...)
```

**SQLite Column Defaults**: When adding NOT NULL columns with defaults to existing tables in SQLite, use `server_default` with raw SQL values:

```python
# Correct - uses server_default for raw SQL
op.add_column(
    "mps_shop",
    sa.Column("stripe_enabled", sa.Boolean(), nullable=False, server_default="1"),
)

# Wrong - default won't work with existing data
op.add_column(
    "mps_shop", 
    sa.Column("stripe_enabled", sa.Boolean(), nullable=False, default=True),
)
```

## Cryptocurrency RPC Access

### Monero Wallet RPC
When investigating or manually testing Monero RPC calls, use digest authentication with these credentials (from Makefile):
- Username: `test_user`
- Password: `test_pass`
- URL: `http://127.0.0.1:18083/json_rpc`

Example curl command with digest auth:
```bash
curl --digest -u "test_user:test_pass" -X POST http://127.0.0.1:18083/json_rpc \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":"0","method":"get_transfer_by_txid","params":{"txid":"transaction_hash_here"}}'
```

### Dogecoin Core RPC
Dogecoin uses basic authentication (from dogecoin.conf):
- Username: `mps_doge_user`
- Password: `change_this_password_in_production`
- URL: `http://127.0.0.1:22555`

## Common Issues and Solutions

### UUID Objects
Always use `uuid_str` when you need a string copy of our identifier. Models inherit `uuid_str` property from `RBase`.

**IMPORTANT**: UUIDs are stored in our database WITHOUT dashes. When querying by ID, remove dashes from our UUID:
- Correct: `WHERE id = '0f92cd2a86f54dc1b98ef5c8b37bc7f8'`
- Wrong: `WHERE id = '0f92cd2a-86f5-4dc1-b98e-f5c8b37bc7f8'`

## Development Standards and Expectations

**CRITICAL WORK ETHIC**: Our user pays significant money for development work and expects thorough, complete solutions. NEVER try to do our minimum or cut corners. When asked to implement features, provide comprehensive, production-ready implementations that consider all aspects of our request.

**CSS LAYOUT REQUIREMENTS — GRID ONLY, NO FLEXBOX, NO EXCEPTIONS**:
This project uses CSS Grid exclusively for layout. **NEVER** write `display: flex`, `display: inline-flex`, `flex:`, `flex-direction`, `flex-wrap`, `justify-content: flex-*`, `align-items: flex-*`, or `flex-grow/shrink/basis`. There is no situation where flexbox is acceptable.

Grid equivalents for the patterns you'd reach for flex:
- **Centering content** (one item dead-center): `display: grid; place-items: center;` (or `display: inline-grid; place-items: center;` for inline-level buttons/badges).
- **Two items, one left one right** (`justify-content: space-between`): `display: grid; grid-template-columns: 1fr auto;` (left item in the `1fr` column, right item in `auto`).
- **Row of items, right-aligned**: don't make the container a grid — set `text-align: right` (or `text-align: end`) and let inline-level children flow/wrap naturally. Or `display: grid; grid-auto-flow: column; grid-auto-columns: max-content; justify-content: end;` if you don't need wrapping.
- **Vertical stack with last item pushed to bottom** (`margin-top: auto` in flex): `display: grid; align-content: space-between;` on the container (works when the container is taller than its content, e.g. inside a `align-items: stretch` parent grid).
- **Equal-height cells in a row**: parent `display: grid; grid-template-columns: repeat(auto-fit, minmax(Npx, 1fr)); align-items: stretch;`.
- **Icon + label header**: `display: grid; grid-template-columns: auto 1fr; align-items: center; gap: var(--space-N);`.

Note: `align-items`, `justify-items`, `align-content`, `justify-content`, `place-items`, `place-content`, `gap` are all **valid in grid context** — only the `flex-*` keyword values (`flex-start`, `flex-end`) and the `flex` shorthand / `flex-direction` / `flex-wrap` / `display: flex` are forbidden. Use `start`/`end`/`center`/`stretch`/`space-between` etc. as the values.

**SHAME LOG — 2026-05-11**: agent blackops shipped `display: inline-flex` on `.mps-button` and several edit-page components (`.edit-status-bar`, `.edit-save-bar`, `.upload-thumbnails-header`, `.edit-card-icon`, h3 headers, `.mps-button-primary`) across multiple commits before fox caught it. All converted to Grid. This rule is non-negotiable; re-read it before touching any CSS.

**DESIGN TOKENS**: All new styles must consume tokens from `tokens.css` — never hardcode colors, spacing, radii, shadows, or font sizes. Use `var(--token-name)` or `var(--token-name, fallback)`. Our token scale uses a 4px spacing base and major third (1.250) type scale.

**STYLEGUIDE**: When creating new UI components (buttons, wells, alerts, layout patterns, etc.), add a live example to `/styleguide` (`make_post_sell/templates/styleguide.j2`). Our styleguide is our single source of truth for our component library. If it's not in our styleguide, it doesn't exist as a pattern.

**CSS MEDIA SIZING**: Never combine `width: 100%` with `max-height` on media elements (img, video). `width: 100%` forces our element to span our full container even when `max-height` constrains our rendered content, creating dead whitespace. Use `width: auto` + `max-width: 100%` + `max-height` instead — our element shrinks to match our actual content aspect ratio within both constraints.

**MOBILE USABILITY**: Never use hover-only interactions (`:hover` to reveal controls, `opacity: 0` with hover reveal, etc.). Mobile/touch devices have no hover state — controls hidden behind hover are invisible and unreachable. All interactive elements (buttons, toggles, links) must be always visible and tappable. Design touch-first, then optionally enhance for desktop hover.

**SPA + NORMAL MODE**: Watch mode uses SPA navigation (`watch.js`) that swaps content without a full page reload. When adding or modifying links, buttons, forms, or any product-specific content on pages that participate in watch mode (content.j2, product.j2), you MUST ensure:
1. **Server-rendered HTML** works for our initial page load (normal mode, no-JS, crawlers)
2. **`updatePageContent()` in watch.js** updates our same element during SPA navigation
3. **Our watch JSON endpoint** (`watch.py`) returns any new data our JS needs

Elements that must stay in sync: CTA edit button, download button, comment form `product_id`, file type/size, description, title, canonical link, related items, comments link count. If you add a new product-specific element, add it to all three layers.

**TESTING INTEGRITY**: NEVER skip, delete, or disable unit tests or integration tests when they break. When tests fail:
1. **FIX OUR TESTS** - Update them to work with new functionality
2. **FIX OUR CODE** - If our tests reveal actual defects, fix our underlying issue
3. **ADD MORE TESTS** - Ensure new functionality is properly covered

Disabling or removing tests weakens our codebase and is unacceptable. Tests are critical safety nets that prevent regressions.

**MANDATORY TEST COVERAGE**: Every new feature, model property, view handler, or form section MUST have tests across all three layers:
- **Unit tests** (`test_models.py`) — Test new model properties, methods, and business logic in isolation using `mock.patch`. No DB required.
- **Integration tests** (`test_integration.py`) — Test interactions between models, especially multi-model workflows (e.g., cart + coupon + gift card).
- **Functional tests** (`test_functional.py`) — Test through our web interface using `webtest.TestApp`. Cover settings form POSTs, page loads, flash messages, and DB state changes.

If a feature touches all three layers (model + view + template), it needs tests in all three files. No exceptions. Untested code is incomplete code.

**AUTO-PUSH**: Commit and push when our work is done — no need to ask fox. If tests were written, they must pass first. If no tests are required (defect fix, config, docs), push immediately after committing. 🔥 == 🔥 — remove all friction.

**DO NOT bump `make_post_sell/GIT_HASH`.** The file is gitignored. `setup.py` rewrites it at install time via `git rev-parse --short HEAD`, so the deployed `/version` hash is always the real HEAD. Manually committing a bumped value just produced churn — each bump's recorded value was one commit behind the actual HEAD it was bumping toward.

## Post-Work Chores

After completing a feature or significant change, always perform these chores before considering our work done:

1. **Tests** — Write unit tests (`test_models.py`), integration tests (`test_integration.py`), and functional tests (`test_functional.py`) covering our new code paths. All three layers are required for new features.
2. **Docs** — Update `docs/architecture.md` (feature toggle matrix, ticket index, diagrams) and `docs/design-system.md` (new components/sections) to reflect our change.
3. **Portal** — Update our marketing site at `~/git/www.makepostsell.com` (feature cards in `index.html`, includes list in `pricing.html`) when a user-facing feature is added.
4. **CLAUDE.md** — Update this file if our change introduces new patterns, form sections, model columns, or conventions that future work needs to know about.
5. **Commit & push** — Per AUTO-PUSH, commit and push when done. No friction. (Do **not** bump GIT_HASH — setup.py handles it at install time.)

## Commit Message Guidelines

**CRITICAL**: Do not include Claude Code attribution in commit messages. Attributing human work to Claude is inappropriate and misrepresents our actual authorship of our code. All code changes should be attributed to our human developer who reviewed, approved, and committed our work.

## Ticket Scoping — One Feature, One Ticket

**Prefer one ticket that fully scopes a feature** over splitting it into `MPS-N`, `MPS-N+1`, `MPS-N+2` phase tickets. Phases inside a single ticket are fine — they let us land work incrementally — but they live in **one document** with **one ticket number**.

Spawn a new ticket only when:
- A "next phase" is a genuinely separate feature with different goals or stakeholders
- A "next phase" is gated on something external (a vendor decision, another team's work)
- A "next phase" has uncertain priority and may never ship

**Why**: splitting one feature across `MPS-24` / `MPS-25` / `MPS-26` fragments commit history, multiplies status pages, and forces a future reader to re-stitch three documents to understand one change. One ticket, with `### Phase 1` / `### Phase 2` headings inside it, is our default. Multiple tickets is our exception, not our pattern.

When in doubt: write it as one ticket. If it grows past ~400 lines or the phases truly drift in goal, split it then — not pre-emptively.

## Capability-Driven Presentation

Follow Russell Ballestrini's capability-driven presentation practice
(russell.ballestrini.net/capability-driven-presentation/). A page need not look
identical across all browsers. Accommodate what our user's browser can do:

1. **Single canonical URI** — one URL serves our content.
2. **Consistent content** — regardless of viewer capabilities.
3. **Graceful enhancement/degradation** — use available capabilities to enhance presentation.

### Our `js-only` / `<noscript>` pattern

```html
<noscript>
  <style>.js-only {display: none;}</style>
</noscript>
```

Apply our `js-only` class to any element that requires JavaScript to function.
When JS is unavailable, these elements hide automatically — our user never sees
a broken control.

### AJAX form submission

Comment forms use progressive enhancement: our form works as a normal POST +
redirect without JS. When JS is available, `comments.js` intercepts our submit,
sends via `fetch()` with `X-Requested-With: XMLHttpRequest`, and inserts our
new comment into our DOM without a page reload (preserving media playback).
Our server returns JSON (HTTP 201) for AJAX requests and falls back to our
normal redirect flow on any error.

## CI/CD Notes

- Build uses `virtualenv-clone` which requires `bin/python` symlink (Python 3.12 `venv` may only create `python3`)
- Our CI creates a symlink before cloning: `test -f env/bin/python || ln -sf python3 env/bin/python`
- When updating Salt states (foxhop-states), always run `salt-run fileserver.update` on our salt master before triggering a deploy — gitfs cache can serve stale files
- MPS uses `caddy_sites.sls` (NOT `sites.sls`) — changes to our uwsgi service template context must be added to **both** files

## Mobile Layout

On mobile (`max-width: 800px`), our product page reorders to single column
with the **buy CTA promoted ABOVE the title** so a phone buyer sees price
+ Add to Cart immediately, before scrolling. The old right-column section
was split into two siblings (`.product-purchase` + `.product-related`,
both also carry `.product-right` so legacy styling still applies). Order:

1. `product-purchase` (order 0) — price, Add to Cart, preview, auction/offer
2. `product-images` (order 1) — title, cover image, thumbnails; sticky in watch mode
3. `product-description` (order 2)
4. `product-comments` (order 3)
5. `product-related` (order 4) — comments link, price history, watch queue, related products

Desktop (`@media (min-width: 960px)`) uses `grid-template-areas`:

```
"images       purchase"
"description  related"
"comments     related"
```

So `product-purchase` sits at the top of the right column next to the
image (where the price has always lived) and `product-related` spans the
rows below.

Cinema mode is a no-op on mobile (gated on `@media (min-width: 800px)`).
The normal watch-mode mobile stack already gives full-viewport-width media
and a consistent section order — the cinema classes exist on the DOM but
match no layout rules below 800px.

Related content on mobile shows only 7 next items (vs 42 on desktop) via `.related-content-overflow` class. A "Comments (N)" anchor link appears on mobile to jump to our comments section below.

## Security

### CWE-407 — Algorithmic Complexity / DoS

**Status**: Partially mitigated. Two distinct attack surfaces.

#### 1. Search keywords + feed endpoints (FIXED — commit f9cbebb)

- `/search?keywords=` — each keyword fired a full table scan; no limit on token count
- Feed endpoints (`/sitemap.xml`, `/rss.xml`, `/atom.xml`, etc.) — unbounded product query
- Fix: keyword count capped, feed queries limited

PoC: `docs/poc-cwe407.py` — tests both surfaces (unauthenticated)

#### 2. Bleach HTML sanitization (FIXED — commit c71fd32)

`bleach.clean()` (via html5lib's tree builder) exhibits **O(2^N)** complexity on crafted HTML.
Measured: N=30 → 1.0s, N=35 → 12.8s. Every +5 chars ≈ 10× slowdown.

**Attack vector**: authenticated user submits crafted markdown with deep nesting (e.g. 35
nested blockquotes = 70 bytes) → html5lib exponential tree reconstruction.

**Fix**: `limit_html_nesting()` in `lib/sanitize_html.py` — flattens any HTML element beyond
depth 20 using `html.parser` (O(N)) before bleach sees it. Wired into `markdown_to_html()`
in `lib/render.py` — single enforcement point for all callers. N=35 drops to 0.04s.

No byte cap — books, long-form content, and deep table-of-contents structures are supported.
Depth 20 covers any legitimate nesting while keeping N well below our exponential zone.

Bleach version: 6.3.0 (html5lib 1.1 vendored inside bleach).
Every webapp calling `bleach.clean(user_html)` is exposed — this is our correct fix.

## Shop Home Layout + Tags (MPS-24)

Optional categorization to cut shopper clicks-to-purchase on shops with a
catalog too large for a flat grid (printableprompts.com has 481 K-1
printables — `Math`, `Seasonal`, `Literacy`, `Novel Studies` are obvious
groups but pre-MPS-24 the home page didn't surface them).

Models: `Tag` (`mps_tag`, shop-scoped) + `ProductTag` association
(`mps_product_tag`). Tags are flat (no tree), many-per-product, unique
per `(shop_id, slug)`. `Product.tags` is an `association_proxy` collection.

Shop columns:
- `home_layout` (Integer, 0=flat / 1=chips / 2=lanes, default 0)
- `home_layout_tag_limit` (Integer, default 8) — caps chips + lanes
- `home_layout_per_lane_limit` (Integer, default 10) — products per lane
- `featured_product_ids_json` (UnicodeText) — optional curated strip
- `tag_stopwords_json` (UnicodeText) — auto-tagger exclusions (Phase 2)

Form section: `home-layout-settings` (`views/shop.py`,
`templates/shop_settings.j2`). Adds the 4 layout columns + featured +
stopwords inputs.

Routes (registered before `shop_slug` catch-all):
- `/s/{shop_id}/tags` — operator bulk tagger (`@shop_editor_required`).
  **Phase 2.8**: tag-focus is AJAX — `tag_bulk.js` intercepts a
  `[data-tag-focus-link]` chip click and fetches `?focus=<slug>` with
  `X-Requested-With`; the view returns JSON
  `{focus, products:[{id,title,url,attached}]}` and the JS swaps the
  `[data-focus-section]` list in place (no full reload — critical on
  large catalogs; the view only loads `all_products` when
  `focus_tag or show_suggestions`). Real HTML5 drag-to-reorder on the
  tag rows POSTs `action=set_order&tag_slugs=…`. No-JS unchanged: the
  `?focus=` link is a real navigation, server still renders the section.
- `/s/{shop_id}/tag/{slug}` — public tag detail page (works without JS)

Tag input on product edit: comma-separated `tags` field on
`product_edit.j2`; server slugifies, dedupes, upserts. **Phase 2.7**:
that field is now the no-JS fallback. With JS, `static/js/product_tags.js`
turns it into a chip editor and POSTs each add/remove to the
`product_tags` route — `/p/{product_id}/tags` (registered before the
`product_slug` catch-all), `@shop_editor_required` + `@trial_active_required`,
`action=add|remove`. AJAX (`X-Requested-With`) → JSON, no full page
reload; plain POST → 302 back to product edit. The JS keeps the hidden
`tags` field in lock-step with the chips so a later full "Save Settings"
is a no-op, never a stale revert. Capability-driven split helper:
`views/__init__.py:is_ajax(request)` (single source of truth;
`shop.py:_is_ajax` delegates to it). Removable chip component:
`.tag-chip-removable` family in `common.css` + `/styleguide#tagchips`.

Home rendering: `views/shop.py:_build_home_layout_context()` builds
`home_chips` (list of `Tag`), `home_lanes` (list of
`{"tag": Tag, "products": [...]}`), `active_tag` (Tag or None),
`filtered_products` (list). `home.j2` + `shop.j2` branch on
`shop.home_layout` to render flat / chips / lanes. `search` view also
receives the chip strip so `/search?keywords=X&tag=Y` works.

Capability-driven: every chip is a real `<a href="?tag=<slug>">` — server
filters on no-JS. With JS, `static/js/tag_filter.js` intercepts clicks
and filters the grid in place via `data-tag-slugs` attribute on
`.serp-item`, zero network cost, fewer clicks to purchase.

**Facet nav (Phase 2.6).** All facet controls live in one Jinja macro
file: `templates/_facet_nav.j2` exports `sidebar()` (desktop `<aside>`)
and `details()` (mobile/tablet `<details>` accordion). Both call the
same internal `facet_form()` so the controls stay identical across
viewports — only the wrapper differs.

Macros take TWO base URLs: `base_url` (slugged shop home, for the
"All" link) and `tag_base` (`request.shop.absolute_url(request,
slug=False)` = `/s/{id}`, no shop slug). Category links MUST use
`{tag_base}/tag/{slug}` so they match the `shop_tag_detail` route
(`/s/{shop_id}/tag/{slug}`). Using the slugged URL produces
`/s/{id}/{shop_slug}/tag/{slug}` which falls through to the
`shop_slug` catch-all and silently renders the shop home instead of
the filtered tag page (the 2.6b→2.6c defect).

Every page that opts into categorization (operator picks `home_layout
>= 1`, or tag detail page) includes BOTH variants in the markup. CSS
hides one per viewport: `details.facet-details` hidden ≥800px, `aside
.facet-nav` hidden <800px. Native HTML, no JS.

Surfaces wired:
- `shop_tag.j2` — tag detail (always shows facets)
- `home.j2` + `shop.j2` — shop home / search when `shop.home_layout >= 1`

Form anatomy: Sort dropdown, Price range (`?price_min=` / `?price_max=`,
parsed by `_price_range_from_request()` → cents, filtered by
`_filter_by_price_range()`), and full Categories list
(`ctx["facet_tags"]` = `tags_by_popularity(...)` with no limit).

**Mobile lanes (Phase 2.6).** When `home_layout == 2` (sectioned lanes),
each `.tag-lane` renders BOTH a horizontal `.tag-lane-grid` of tiles AND
a vertical `.serp-list.tag-lane-rows` of `.serp-list-row` items with
6-sentence excerpts. CSS toggles: tiles ≥800px, SERP rows <800px. The
mobile/tablet experience now shows description + title + price per
product instead of a Netflix-style swipe row that hides the description.

**SERP excerpt.** `Product.excerpt_sentences(n=6, max_chars=1500)`
returns six sentences, markdown-stripped, with a 1500-char safety cap
for descriptions without terminators. `Product.excerpt()` (char-based)
and `Product.excerpt_sentences()` both consume the module-level
`_strip_markdown()` helper for one source of truth.

Phase 2 (shipped): deterministic title-plus-description auto-tagger
in `lib/tag_suggest.py`. Title tokens weight × 3, description × 1
(capped at 100 unique tokens per product). Pipeline: tokenize →
~200-word English + per-shop stopwords → suffix-strip stem →
form **bigrams** from adjacent non-stopword tokens (`write room`,
`first grade`, `valentine day` — phrases get 2× unigram weight) →
group by stem/bigram → drop keys matching existing tag slugs →
**`max_share` filter** drops keys appearing in >40% of products as
shop vocabulary → **`min_title_share` filter** drops keys appearing
in <30% of carrier products' *titles* (kills description-only noise
like `versions`, `engaged`, `offered`, `during`) → rank by product
count desc → label = most frequent original word/phrase. Returns
`(clusters, filtered_count)`. Surface: button on `/s/{id}/tags` →
"Suggested categories" well with one-click Apply / Dismiss per
cluster (`action=apply_suggestion` / `action=dismiss_suggestion`).
URL knobs: `?max_share=0.3` (stricter shop-vocab cut), `?max_share=1`
(disable), `?min_title=0.5` (stricter title-required), `?min_title=0`
(disable), `?bigrams=0` (disable phrases), `?top_n=200` (show more).
CLI: `python -m make_post_sell.scripts.backfill_tags
data/development.ini --shop=<id> [--max-share=0.4]
[--min-title-share=0.3] [--no-bigrams] [--apply]`. **Never
auto-commits** — operator approves every cluster.

Phase 3 (this ticket, gated): ML categorization via uncloseai endpoint
behind `app.features.ml_categorization.enabled` kill switch (mirror MPS-22).
Suggest-then-approve; never auto-commit.

## Auction & Make-an-Offer (MPS-20 + MPS-21)

Both ride on `Product.pricing_mode` (Integer, 0=fixed, 1=auction,
2=auction+buy_now, 3=offer, 4=offer+buy_now). Owner flips via product
edit form; the system creates a draft `MpsAuction` row when flipping
into auction mode.

Tables: `mps_auction`, `mps_bid`, `mps_auction_watcher`, `mps_offer`,
`mps_offer_event`, `mps_cart_auction`, `mps_cart_offer`.

Cart integration overrides `Cart.total_price_in_cents` when a
`cart_auction` or `cart_offer` association exists — pays the winning
bid or accepted offer amount instead of `Product.price_in_cents`.

State transitions are driven by cron:
- `make_post_sell.scripts.auction_tick` (every minute) —
  SCHEDULED→ACTIVE on start_timestamp pass, ACTIVE→ENDED on
  end_timestamp pass.
- `make_post_sell.scripts.offer_tick` (every 15min) —
  PENDING/COUNTERED→EXPIRED past expires_timestamp.

Form sections:
- `pricing_mode` + `allow_offers` on product edit.
- `offer-settings` on shop settings (7 fields:
  offer_enabled, auto_accept_threshold_pct, auto_decline_threshold_pct,
  offer_min, offer_expiration_hours, offer_max_rounds,
  offer_min_buyer_account_age_hours).

Routes (registered before `product_slug` / `shop_slug` catch-alls):
- `/a/{auction_id}` + `/a/{id}.json` + `/a/{id}/{events,bid,buy-now,watch,checkout}`
- `/p/{product_id}/offer` (open) + `/o/{offer_id}` + `/o/{id}/{events,counter,accept,decline,withdraw,checkout}`
- `/s/{shop_id}/offers` — operator inbox (`@shop_editor_required`), linked from `/actions/view`

Live updates use **bounded SSE** (`lib/sse.py` → `sse_response` /
`event_stream`): `/o/{id}/events` (buyer/seller only) and `/a/{id}/events`
(public) stream `text/event-stream`, poll the row ~every 1.5s, emit on
state change, heartbeat, then close after ~25s so `EventSource`
reconnects. uWSGI is sync (~16 threads) so a truly long-lived SSE would
starve the pool — hence "bounded". The generator must use its **own**
short-lived DB session per poll (`request.registry["dbsession_factory"]`),
**not** `request.dbsession` (pyramid_tm has already closed it by the time
the streaming generator runs). Timings: `app.sse.hold_seconds` /
`app.sse.poll_interval_seconds` settings (tiny in `test.ini`). `offer.js`
reloads on a state change; `auction.js` calls `applyState()` per frame and
falls back to 5s polling of `/a/{id}.json` where `EventSource` is absent.
Caddy auto-detects `text/event-stream` and stops buffering — no Salt
change needed.

Offer/auction POST routes are **capability-driven**: a plain browser
submit gets a flash + `302` redirect; an AJAX submit (`X-Requested-With:
XMLHttpRequest`) gets JSON. `static/js/offer.js` + `auction.js` are the
enhancement layers. `offer.j2` shows a `.offer-state-notice` banner so
the state is clear without a flash.

Identity/privacy: never render a user's email in offer/auction UI. Show
`User.display_name` (= the public `name` handle; **`full_name` is
private**) linked to `/profile/{handle}`. The public profile page
(`views/user.py:user_profile`, route `user_profile` → `/profile/{name}`)
reveals the email only to the user themselves, or to a shop owner/editor
viewing in that shop's context (`?shop={shop_id}`) when the profile user
has transacted there. `User.gravatar_url(size)` forces an identicon
unless the user opted into Gravatar (`user.gravatar`).

See `docs/auction-house.md` and `docs/make-offer.md` for full state
machines and architecture.

### Actions hub (`/actions/view`)

`actions_view.j2` is one flat `.action-button-grid` of `.mps-button` links
inside an `.action-columns` well — single column on narrow viewports,
**at most two equal columns** at ≥720px (`grid-template-columns: 1fr 1fr`),
auto-flow row-by-row so the columns stay balanced. No `<br>` spacers. The
grid resets `.mps-button`'s `min-width: 100%` + auto margins (they fight
track sizing and eat the `gap`). Add a new operator shortcut as another
`<a class="mps-button product-edit-button">` in that grid.

## Feature Kill Switches (MPS-22)

Global feature flags live in `data/development.ini` (and override via env var
in `~/git/foxhop-pillar/uwsgi/makepostsell/init.sls` for prod). Pattern mirrors
`app.features.popout_player.enabled` — read via reified request property.

| Flag | Property | Default | Status |
|------|----------|---------|--------|
| `app.features.popout_player.enabled` | `request.popout_player_enabled` | True | Working |
| `app.features.karaoke.enabled` | `request.karaoke_enabled` | **False** | Broken (MPS-18) |
| `app.features.torrent.enabled` | `request.torrent_enabled` | **False** | Broken (MPS-19) |

When a flag is off:
1. Templates wrap UI in `{% if request.X_enabled %}` — section hidden
2. Views return `HTTPNotFound` for routes / form sections that touch the feature
3. Views set context values for that feature to None / "" / False
4. Backfill paths skip work
5. Spawn paths (karaoke detached child, torrent generation) bail early

`test.ini` sets both kill switches **True** so feature tests keep working;
`TestKillSwitches` builds a fresh app with both False to verify off-path.

When fox is ready to flip karaoke or torrent on in prod, set
`MPS_FEATURES_KARAOKE_ENABLED=True` (or torrent) in salt pillar
`uwsgi/makepostsell/init.sls`, then deploy.

## Operation Voyeur

**All comms are public** from 2026-03-29. Assume every terminal session and output is observed. NEVER display secrets to stdout. NEVER pass secrets as CLI args (`ps aux` sees them). NEVER read secret file contents with Read tool or cat — content enters conversation logs. **Path is fine. Content is not.** Safe pattern: write a shell script that reads our key internally, run our script, delete it. Credential locations (paths only): GitLab `~/.config/gitlab/token`, Namecheap `~/.namecheap/api.key`, ImprovMX `~/.improvmx/api.key`.

### Never broad-grep config files

Config files live next to secrets. `production.ini`, `development.ini`, `.env`, `vars.sh`, `pillar/*.sls`, `/etc/*.conf` — all mix settings with credentials. A pattern like `grep -iE 'bucket|s3|region|access_key|endpoint'` looks narrow but matches `secret_key` because the file itself groups related keys together — one match pulls every neighbor into a log line.

Rules:
- **Grep for an exact key name, not a category.** `grep '^app.bucket.secure_uploads.region' production.ini` ≠ `grep -iE 'bucket|region'`.
- **Never include `secret`, `password`, `key`, `token`, `access`, `credential`, or `auth` in a grep alternation run against a config file.** If we need to check that a secret key IS set, `grep -c '^app.bucket.secure_uploads.secret_key' file` returns 0 or 1 — value stays in file.
- **Prefer `test -f` / `wc -l` / key-exists checks** over anything that prints file content. Path is fine. Content is not.
- If we accidentally pull a secret into terminal or transcript: **flag it immediately** and tell fox to rotate. Never pretend it didn't happen.

## Style

- Prefer "our" for shared things; "a" when something is one of many; avoid "the" — it implies fixed, singular ownership. Most teams and systems are fluid and ever-changing, like water.
- **Never use "AI" — always say "machine learning."** We grow machine learning, not "AI." This term is forbidden in all permacomputer discourse, marketing, & documentation.

## Prime Mission — Balance All Workstations

Every node is a workstation. Every edge is a queue. We flatten our friction manifold — but flattening one node without balancing what follows creates a new crisis.

**MOAD-0001 & MOAD-0005 are coupled.** Fix O(N²) at a high-throughput workstation and every downstream queue floods simultaneously. Solving one defect creates the other if we do not stage capacity first.

**3 drivers. 3 million people.** Fix the dispatch. Stage the drivers first.

- **Workaholic node** (high betweenness + high speedup): IS our bottleneck. Unblock without staging = collapse.
- **Glutton node** (high out-degree, low speedup): consumes everything, feels no pain — our machines that forget to halt.
- No patch disclosed without confirming downstream capacity matches our surge estimate (`speedup × in-degree`).
- Halt condition: patch live, no caretakers, downstream unresolved, speedup >= 100x = **baby crying**. Assign team first.

Full factory model & live DAG: `~/git/undefect.com/generate_dag.py`.
Shard source of truth: `~/git/unsandbox.com/blackops/BLACKOPS.md`.

### Eight Forms of Capital — Stewardship Check

Every feature, patch, & system decision touches at least one of our 8 capital queues (Roland & Landua, via unturf.com/eight-forms-of-capital/):

Living · Material · Financial · Intellectual · Experiential · Social · Cultural · Spiritual

Before shipping: does this drain a workaholic to feed a glutton? Does it route away from a food desert? Does it grow financial capital at the expense of living capital? If yes — stop. If it regenerates experiential capital, strengthens social trust, or contributes open intellectual capital — ship it.

Platform tax = O(N²) friction in our exchange layer. Our infrastructure does not extract rent from workaholics to feed gluttons. That is our obligation as permacomputer stewards. Full ledger: `~/git/unsandbox.com/blackops/BLACKOPS.md`.